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have been suggested as a possible tool in training spatial knowledge. However, little 
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group and trained to a set standard rather than to a time. The second experiment also had 
a map-only and VE group, but trained one hour with a low fidelity map (1:24,000 scale as 
compared to 1:5,000 scale in earlier experiments). Measures were taken of landmark, 
route, and survey knowledge. 

The results suggest that, (1) subjects who trained-to-standard using a VE 
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I. INTRODUCTION 
A. PROBLEM STATEMENT 


This thesis will attempt to answer a simple question: What is the value added (if 
any) of using a virtual environment (VE) as an augmentation to map study for terrain 
familiarization. The thesis will analyze and compare an individual’s acquisition of spatial 
knowledge from a high-fidelity virtual representation of a specific area to those trained 
with traditional map techniques (See Chapter III Training). The research intends to 
investigate unexplained questions from earlier research and to study the validity of using 
virtual environments to acquire spatial knowledge of an area of real world terrain. If 
proved valid, future research could investigate the optimal level of fidelity, optimal level of 
immersion, and optimal interface control devices maximizing knowledge acquisition. If 


disproved, efforts in this area would be better focused on other types of tasks. 


B. MOTIVATION 
1. Army and DOD Relevance 


The Department of Defense (DOD) is actively looking for ways to train personnel 
while conserving precious dollars and improving safety and performance. Many of DOD's 
explorations have involved Computer Based Trainers (CBTs). Virtual environment 
trainers and simulators have proven successful for combat aircraft simulators [ANGI 
93][CRAN 93] and armored vehicle crews [BOLD 85][BROW 88]. They may also prove 
a useful technique for dismounted soldiers for both training and mission rehearsal. Such a 


system is hoped to give soldiers performing a mission better situational awareness. 


General William Hartzog, commander of Training Command described situational 
awareness on the battlefield to be: Knowing where you are, knowing the location of 
friendly forces, and knowing the location of enemy forces. Critical to knowing where you 
are is knowledge of the ground around you [BANK 97]. This mirrors the advice of Sun 
Tzu some two thousand years before: “We are not fit to lead an Army on the march unless 


we are familiar with the face of the country - it’s mountains and forests, its pitfalls and 


precipices, its marshes and swamps [SUN 83].” Any system that allows soldiers to better 
know an area before walking the ground for the first time would greatly influence 
operations. The Army’s doctrine states that terrain is clearly not neutral in any conflict. 
The terrain provides an advantage to the side that better understands the features, 
advantages, and limitations of a given area and plans accordingly [FM10 93]. But as the 
American Army changes from a forward deployed force to a power projection one [CJCS 
98], soldiers and leaders are afforded less opportunity to train or maneuver on the ground 
they may later have to fight over. By virtue of the enhanced movement ability in computer 
environments, individuals may be able to explore more of an area in a VE than if actually 
walking the ground. This would allow soldiers to train faster than before in some cases 
through time-compressed training. It may also allow them to become familiar with an area 
before actually arriving there in person. A VE would also allow users to train in areas that 
may not be accessible due to hostile forces or prohibitive travel costs. This is not to 
suppose that VEs could entirely replace real world ground training. No VE in the near 
future can replicate desert training down to the sand in your boots. The best VEs and 
simulators augment real world training, they do not replace it [SCHW 86][САТЕ 
87][BROW 88]. 


One foundation of military forces is conducting rehearsals prior to performing a 
mission. The military has a long history of creating mockups as part of these rehearsals 
[GLIN 95] [MCRA 95] [FINN 97] [AMER 98]. These aids may be as simple as some 
circles drawn in the dirt to elaborate buildings and structures built for some missions. The 
one thing the military has not been able to do is make the terrain at the rehearsal site 
accurately match that found at the mission site. VEs may provide a cost-effective way of 
doing just that. While they would not replace the need for actual building models, they can 
allow soldiers to better prepare for the ground around target areas. As a mission rehearsal 
tool, they would allow soldiers to rehearse their movement to and from the target area, 


and perhaps explore alternative routes to ensure the optimal selection. 


However, just because a training tool involves a computer, this does not 
automatically make in better than alternatives [KEAR 83]. Unlike the Army's Simulation 
Networking (SIMNET) program [SCHW 86][BROW 88] [BURN 90] [ANGI 93], there 


has been little research published or conducted into the ability of VEs to impart spatial 
knowledge of an area. The Air Force recently used VE systems in Kosovo to conduct 
mission rehearsal in a system strictly designed to give terrain awareness [THOM 99]. 
While this may seem useful based on “common sense,” research has largely not validated 
the concept. If soldiers and pilots train and prepare more effectively (as defined by better 
task performance) with maps than with VEs, then it is a waste of time and resources to 
develop such systems for this purpose. Even if performance is better, the gain may not 
outweigh the increased cost in time and money such a system imposes. Another potential 
pitfall is the lack of an ideal interface to train dismounted soldiers. While vehicles confine 
soldiers to the vehicle’s limitations, individual soldiers are much more flexible in their 
actions. VE researchers have yet to build an interface that allows the soldier to crawl, 
walk, and run through a computer environment as naturally as he can in the real one 
[DARK 97]. This lack of ideal interface has not precluded the inclusion of dismounted 
soldiers in models however [GOUR 99]. 


VEs may not be useful in all the ways proponents envision. They may not prove 
useful as mission rehearsal tools, or as route rehearsal tools. They may not assist in 
acquiring spatial knowledge of an area. They might not even be useful as a tool to teach 
general navigation skills. As with any computer training aid, they may not actually be any 
better than the methods currently in use [KEAR 83]. Without knowing the effects of 
computer training, both positive and negative, the Army cannot make accurate decisions 
on their implementation within the force. With today’s declining budgets, the Armed 


Forces can ill-afford to waste money on hunches or intuition. 


Before actually spending money to contract for such a system, it is critical to 
understand the desired effects and actual achievable benefits derivable from VE training 
systems. Without this, it is impossible to create an accurate set of parameters for the 
models. By doing this research, the Army can determine if such a system is beneficial and 
for what purposes (area familiarization, route rehearsal, mission rehearsal, or general land 
navigation training). It will help prevent the wasting of time and money on potentially 


expensive yet unproductive systems that do not achieve the desired benefit. It will also 


provide a common set of standards which the Army, civilian contractors, and 


programmers can consult during the development of such systems. 


2. Important Applications for Spatial Knowledge Acquisition 


A wide range of jobs and activities rely upon spatial knowledge acquisition. Most 
people would readily recognize the need for emergency personnel to use spatial 
knowledge to pick the quickest route to arrive at the scene of an accident. People easily 
see the requirements of police, firemen, and paramedics to possess this type of knowledge. 
Another good example is cab drivers using their knowledge to avoid delays. Some of the 
most mundane tasks also require this skill; finding specific items in a supermarket, for 


example, or in a stadium, locating the nearest restroom and then finding your seat again. 


The military, more than most, relies heavily on spatial knowledge acquisition. 
Soldiers and other military services use their knowledge of the environment to assist them 
in accomplishing their mission, be that to attack, retreat, defend, or just observe [GOER 
98b]. This requirement is not limited to foot soldiers. As seen recently in the Kosovo air 
war, pilots too need accurate knowledge of their operating environment to successfully 


accomplish their task in a safe and efficient manner. 


In short, any activity that requires movement through space or through complex 
environments does in fact require spatial knowledge acquisition. These can be military or 


civilian but they are pervasive throughout our lives. 
3. Spatial Knowledge Acquisition as a Proven Concept 


Researchers have extensively studied spatial knowledge acquisition because it is 
such a prevalent and important requirement throughout the scope of human activity. The 
most accepted model of human ability to acquire detailed spatial knowledge is provided by 
Thorndyke who created a simplified model to explain the process [THOR 80](see Chapter 
II Spatial Knowledge Acquisition). Since we know that humans can acquire spatial 
knowledge about real world areas, researchers next studied if humans could do the same 
with VEs [GILL 97] [RUDD 98]. Following this, they also examined if this spatial 
knowledge could be transferred from the VE to the real world area [WITM 95] [BLIS 97] 


[DARK 98] [WALL 98]. Because this concept was proven for VEs in general and the 
transfer from VE to the real world in man-made indoor environments, it was natural to 
next explore if this previous research holds valid when moved to natural unstructured 


outdoor environments. 
4. Existing Research Shortcomings 


While Chapter II discusses much of the existing background research for the 
thesis, several points are highlighted here. Many problems exist with current CBTs 
because there is a lack of research to back up their purpose. There is little proof that using 
certain CBTs is any better, if not worse, than current techniques. Furthermore, where 
there is proof, it is often the cost savings, not the benefit gain that is the advantage. For 
example, only about 59% of tasks trained in a flight simulator carry over into the real 
world, but at 10% of the cost of operating the actual plane [ANGI 93]. Many groups seem 
eager to jump onto the technology bandwagon and do not first see whether using the CBT 
results in “better” task performance. Users should justify any shift towards CBT in terms 


of cost or performance gain. 


As computer-processing power has improved, so too has the achievable level of 
fidelity in VEs. They can be more accurate and more realistic looking than in past years. 
There has been much effort spent on increasing fidelity in the models, but little research to 
show how this increased fidelity affects performance. Determining the optimal level of 
fidelity will be critical for building future VEs that are efficient for training skills. More 
research into the area of fidelity is needed. 

Because the study of transfer of spatial knowledge from VEs to the real world is 
relatively new, there are many questions remaining unanswered. The most important is 
how to maximize the transfer. In an ideal case, if a person performs the task in a VE, it 
would have the same learning affect as if he did it in the real world. Questions as to the 
ideal interface, display, fidelity, and movement within the virtual environment remain 
unanswered. VEs provide the user with the ability to do things impossible in the real world 
(e.g. travel at any speed, fly, and teleportation) but the most effective combination of these 


tools is unknown. Do these tools aid learning, or are they distractions? Finally, there is a 


lack of research extending the transfer of spatial knowledge from VEs to the real world in 


natural environments. This thesis intends to address that issue. 


C. THESIS ORGANIZATION 


The thesis is organized as follows: Chapter II provides background of the 
military’s use of simulations, spatial knowledge acquisition, learning techniques, previous 
studies on the use of VEs, and the work of prior students that this thesis builds upon. 
Chapter III discusses the VE that MAJ Simon Goerger developed and used in this thesis’ 
experiments. It also presents the details of all the tests, tools, and methods used in the two 
experiments. This section also discusses the two different experiments conducted as part 
of the thesis and their respective methodologies. Chapter IV analyzes the data collected 
from the two experiments and discusses the results. It also compares the results to the 
original Goerger experiment. Chapter V compares the findings of the two experiments and 
the original Goerger study, discusses their impact, and lays out future areas of interest for 


follow-on research. 


The thesis includes numerous appendices that show the maps, participant 
instructions, and experiment’s timelines. Information concerning route difficulties and 
common land navigational terms are also included as appendices. In addition the raw data 


from the experiments is included. 


Il. BACKGROUND AND PREVIOUS WORK 


A. THE MILITARY AND MODELS/SIMULATIONS 
1. Historical use of Simulations for Training 


The military use of wargames or other simulations for training is only slightly 
younger than the history of organized warfare itself. Archeologists have discovered groups 
of miniature Sumerian soldiers in formation, as well as later Egyptian units [PATR 77]. 
While these may indeed have started as mere toys for the nobility, at some point the play 
with these soldiers became stylized, evolving into games such as Go and Chess. While the 
value of Chess as a direct simulation has obviously faded far in the past, the spirit lives on. 
In 1780, Helwig created the first wargame to expand beyond the set “chess-style” 
boundaries. It included 1666 squares representing different terrain, and players had 120 
units per side, including cavalry, forts, and artillery. 

In 1811, Herr Von Reisswitz did away with the idea of a game board entirely, 
introducing a game played on a sand table, where terrain matched a map at 1:2373 scale. 
His son later transformed the game to maps at 1:8,000 scale and in 1824 introduced it to 
then Chief of Staff of the Prussian Army General Von Muffling, calling it Kriegspiel (war 
play). After the demonstration, Von Muffling, who had been unenthusiastic before, 
exclaimed “It is not a game at all, it’s training for war! I shall recommend it most 
emphatically to the whole army [PATR 77].” One of the major evolvements of the 
Reisswitz wargame was the use of umpires. These neutral parties were used to provide 
fog-of-war and resolve disputes. Wargames became even more popular in Prussia under 
Von Molke. After the stunning German victories in 1870, many nations rushed to include 
wargaming into their training programs as well. The Japanese in particular credit their 
victory in the Russo-Japanese war of 1904 in part to their wargaming. 

Within the United States, the military first picked up on wargaming in 1867 and by 
1882 the American Army produced it’s own wargame, The American Kriegspiel, A Game 
for Practicing the Art of War Upon a Topographical Map. Later the American Army 
moved to force-on-force engagements that were more involved than the staff room 


training STX/CTXs (situational training exercises/combat training exercises). Here units 


maneuvered, dug in, and actually did everything but shoot. Historical events such as the 
Louisiana Maneuvers prior to World War II and the military’s annual REFORGER 
exercises in the 1970/1980's are of this type. The US Army even developed a manual for 
its umpires, FM105-5 Maneuver Control, to standardize rules about casualties, advance 
rates, etc. One problem with this system is that often the training would break down into 
“I shot you, no you didn’t” disagreements more reminiscent of children playing than an 
army training. 

With the progress of technology, especially the introduction of the MILES 
(Multiple Integrated Laser Engagement System) system, the Army was able to remove the 
umpire from direct involvement in actual engagement training. This made the training 
more realistic and solved the “I shot you” problem. Now, the sensors determine if a hit 
was scored or not. It allowed for better training at night where before the umpires had 
been hampered. Different laser signatures were used for different weapons so that a rifle 
could not destroy a tank. The rise of SIMNET (see Section A.3) allowed the fighting to be 
removed from the field and into computers. 

Currently, the US Army defines three different types of models and simulations 
that support training: live, constructive, and virtual. Live simulations are those that 
involve personnel and vehicles in the field. These may involve umpires or the MILES 
system to assist in results [DMSO 93]. Constructive simulations hearken back to the 
original wargames, and are used in staff planning and training exercises. They are useful in 
simulating large conflicts across a region, often involving the entire theater [DMSO 93]. 
Currently the Army uses such systems for its BCTP (Battle Command Training Program) 
and its WARFIGHTER exercises (evaluation exercises for Divisional Staffs). The final 
classification of simulation is Virtual. The SIMNET and Close Combat Tactical Trainer 
(CCTT) programs fall into this category (see Section A.3). 

Why do units conduct simulations? General Gorman, the former commander 8" 
Infantry Division, said “The first battle of most wars fought by the Army of the United 
States was a disaster: a costly defeat or a Pyrrhic victory." [GORM 90] He added that 
simulations that can be rehearsed repeatedly to increase combat readiness, refine combat 


skills, and protect the force before actual combat are invaluable. The goal is to impart 


combat experience without the actual dangers of combat. In studies of real combat results, 
inexperienced soldiers and pilots suffered higher causality rates, but as they became more 
experienced, their survivability improved. The use of simulations for training can have 
dramatic impact upon battlefield results. For example, during the Vietnam War, one US 
fighter was lost for every two North Vietnamese fighters. This ratio improved to 12.5 to 1 
after the Navy implemented its TOP GUN program, which was an engagement simulation 
[GORM 90]. The best reason behind using simulations is that “The bottom line is it 
prepares us to be the best we can.” [BELC 99] 


2. Model Usage for Mission Planing and Preparation 

Throughout military history, forces have incorporated detailed rehearsals into their 
preparation for coming battles. The US Army’s Ranger Handbook, the bible of its infantry 
forces, states “They [rehearsals] are essential to ensure complete coordination and 
subordinate understanding.” The handbook adds that they should be conducted in an area 
as similar to the objective as possible. The rehearsals should include brief-backs over sand 
tables, or sketches [RHB 95]. While many of these rehearsals may involve merely simple 
checklists, or talk-through of the planned battle, others are more detailed. These may 
involve the use of sand tables, map-boards, dioramas, simple dirt maps, or even elaborate 
mock-ups, to better picture the terrain, as well as the individual unit’s locations in time 
and space. For many operations, the military would actually build a life size model of the 
target so units could practice moving about the ground. This was what the Israeli Army 
did in its preparation for the Entebbe airport raid. They laid out the terminal using metal 
poles and burlap for walls to exact dimensions [MCRA 95]. The troops were also shown 
video footage of the terminal area (taken from a soldier’s personal vacation footage). 

Similar missions did not always have similar preparation time but always included 
at least some form of rehearsal. Consider the US Ranger raid on Cabanatuan in January 
1945 in the Philippines in WWII and the Son Tay raid during the Vietnam War. The 
rangers liberating the Philippines had crude dirt-maps and sand table briefings conducted 
at patrol bases within gunshot of the enemy. They had less than three days to prepare their 
mission and much of that time was spent near the prison camp some 70 miles behind 


enemy lines. Despite this, their final preparations involved briefing all soldiers over the 


camp layout and objectives based on the most recently scouted information. The forces 
raiding Son Tay trained extensively for months on elaborate mock-ups [MCRA 95]. The 
entire complex was replicated in exacting detail for rehearsals, as well as all other aspects. 

During World War II, the Army Air Corps prepared its pilots for air raid missions 
over Japan by showing them films of the precise routes they would be flying over enemy 
territory. These films were not produced from satellite imagery or over flights by 
American reconnaissance aircraft. Instead, the films were produced by the Air Corps film 
and production unit stationed in Hollywood, California [AMER 98]. The production unit 
built a model of Japan using over fifty ten foot square platforms, tons of plywood, 
modeling clay, burlap, and paint. Using reconnaissance photos of the island, crews worked 
twenty-four hours a day for weeks, expending thousands of man-hours, to construct and 
paint an exacting replica of the island so that camera crews could film bombing routes for 
pilots. In order to maintain security, the model was built and filmed entirely on a single 
sound stage that was placed off limits to everyone except the personnel working on the 
project. Pilots routinely commented on how easy it was to recognize the terrain as they 
flew their missions because it was as if they had been there before. 

Many of the models used in recent military operations came from the CIA’s 
Modeling Shop [FINN97]. This group specializes in creating detailed replicas of various 
sites and structures. They helped in the Son Tay raid in 1970, and made replicas of the US 
Embassy in Iran and the Kremlin in Moscow. The hours and expense in making these 
models was immense but considered valuable based on the requirement for detailed 
exposure to the environment prior to the mission. However, these models, in all their 
detail, lacked the ability to actually walk into the structure and move around. The 
Modeling Shop shifted to making computer generated models in 1997 [FINN 97]. 


3. The Military and VEs 


The current military is smaller than at any time since WWII and has shifted from a 
forward-deployed force to a power projection one [DMSO 93]. The reduction in size and 
shift in missions has forced the military to change its training and doctrine. The military 
has become very interested in using simulations and VE in particular to augment 


traditional training. 


As stated above, the military had always found simulations useful, but the advent 
of VEs allowed them to take advantage of the new technology. Among the first systems 
built were single vehicle trainers. Flight simulators were developed to allow pilots to train 
at a fraction of the cost of actually flying jets. The Army developed the UCOFT system 
(Unit Conduct of Fire Trainer), which was designed to train tank crews in gunnery 
operations. The fact that the UCOFT provided improvements in gunner proficiency was 


seen as validation of the use of simulators for non-aerial vehicles [BOLD 85]. 


The Army places great emphasis on training as combined arms or “training as we 
fight" [FM25 88]. While single vehicle simulators worked, the military fights as units and 
teams, so the military was looking for systems that would allow these groups to be 
brought together for collective traming. Networking simulators were the result of Defense 
Advanced Research Project’s Agency (DARPA) sponsored technology demonstration 
projects [DMSO 93] and were the obvious answer to the collective training problem. 
Among the products were those for the Army’s tank and infantry fighting vehicle 
(SIMNET). Later systems were added for A-10s and attack helicopters. These systems 
brought the individual vehicle simulators into one virtual world and allowed them to 
maneuver together in real-time. This allowed the units to practice together in the 


simulators making them much more useful. 


The Army’s latest development is the Close Combat Tactical Trainer (CCTT). 
Using high fidelity manned modules, it represents combat between armored vehicles and 
dismounted infantry. The system is not designed to be an individual skills trainer [DMSO 
93], rather one that trains teams, platoons, and companies. The goal of the system is to 
allow the platoon through task force/battalion level to train on collective tasks cheaper and 
more safely than they can in the field [GOUR 99]. While not designed to replace field 
training, it will be used to augment it, as training dollars become short. The envisioned 
advantage over the current SIMNET system is a much higher fidelity level, allowing for 
the inclusion of realistic smoke, noise, and debris effects [DMSO 93]. This will make the 
training more realistic and follow the guidance of FM25-100 Training the Force, which 
says all training should strive to be as realistic as possible [FM25 88]. The goal is that use 


of this system, while saving money, will not reduce the realistic training conditions. Note 


that the DOD has no evidence that the inclusion of these effects will result in better 
transfer of task learning, nor in increased performance. This follow-on to SIMNET is 


hoped to be fully functional by 2003. 


Another system of note currently in use is the Topscene system. “ Topscene is an 
aircraft simulation using real-time and real imagery to conduct mission planning, mission 
rehearsal, and pilot training”[DMSO 93]. The system’s primary use has been in visualizing 
threat radiuses, planning egress routes, and terrain masking. It is not a combat simulation 
but more of a planning simulation. Recently, pilots in Kosovo used TOPSCENE during 
the conflict prior to conducting their missions [THOM 99]. While there is no statistical 
data to back their opinions up, pilots say it helps them identify landmarks and assists in 
mission rehearsal [THOM 99]. 


B. SPATIAL KNOWLEDGE ACQUISITION 

Spatial knowledge or spatial cognition is a mental representation of a real or 
virtual environment [WICK 92]. Thorndyke theorized that there were three levels of 
spatial knowledge: Landmark, Route, and Survey Knowledge [THOR 80]. 


Landmark knowledge is defined as being able to recognize distinctive features or 
locations located at a specific location in an area. Landmark knowledge is the ability to 
memorize features in an environment, such as specific hills, road intersections, or 
buildings. Later, this knowledge is able to be recalled allowing individuals to quickly 
recognize the feature or location when they see it. Landmark knowledge is most often 
acquired by direct exposure to an environment, though it is also possible to gain it through 
study of a map or photograph of the area. Individuals successfully demonstrate landmark 
knowledge by their ability to recognize distinct locations or unique objects within an area 
[DARK 95] [THOR 80]. Figure 2.1 shows an example of an individual with landmarks 
knowledge of an area. The subject would recognize that the area has three distinct features 
X, Y, and Z and will recognize these features if he sees them. The subject would not 


necessarily know any information concerning the relative positions of these features. 





Figure 2.1. Landmark Knowledge 


Route knowledge is identified as the ability to navigate along a route or path 
between landmarks or distant locations [GOLL 91]. It expands the recognizing of specific 
areas from landmark knowledge into a more complex arrangement of linking those areas 
by a path or route. Route knowledge is derived from an egocentric (inside-out) viewpoint 
and is characterized by being able to move from one landmark to the next following a 
prescribed path. Like landmark knowledge, route knowledge is gained through repeated 
exposure to an environment or through the study of a map or overhead photograph. The 
exposure to a video route of the environment has also been shown to develop this 
knowledge [GOLD 82]. This was also shown during WWII by the Army Air Corps’ 
produced “films” of bomber routes over Japan. The pilots credited these films, which 
showed the pilots the routes they would follow on their attack, with making it easy to 
confirm they were in the right place. The fact that these films were not of the actual 
mainland but were created from an extensive model had no effect on the results 
[AMER98]. Figure 2.2 shows an example of an individual with route knowledge of an 
area. The subject not only has knowledge of certain landmarks and distinctive features (X, 
Y, and Z), but they also have the knowledge to traverse from one landmark to another 


along at least one set path (the route from X to Y and then to Z). 





Figure 2.2. Route Knowledge 
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Route Knowledge does not necessarily confer the ability to conduct the reverse of 
the learned route, nor does it mean they may know of alternate routes or shortcuts. It does 
not imply that the subject knows the orientation of Y from X, just that they know a route 
or path on how to travel from one to the other [KOH 97]. 

The final and highest level of spatial knowledge is survey (or configurational) 
knowledge. It represents a map—like or top down mental encoding of the environment and 
is based on an exocentric (outside-in) viewpoint. In survey knowledge an individual can 
not only recognize specific locations or landmarks, but can accurately place them in his 
environment even if he cannot see them. Individuals can also traverse throughout their 
area without having to pre-plan the exact route because they know the layout of the 
region [BANK 97]. Because of its exocentric nature, survey knowledge is most often 
gained through map or aerial photograph study. Extensive exposure to an area has also 
been demonstrated to develop this knowledge [THOR 80]. Figure 2.3 shows an example 
of an individual with survey knowledge of an area. The subject can describe the relative 
distances and locations of all major objects in the area in relation to the subject and each 
other. The subject at d can tell how far he is from X, Y and Z, even if he cannot see them. 


He can also plot a route to any feature without needing prior study. 





Figure 2.3. Survey Knowledge 


Thorndyke’s theory on the process by which humans acquire spatial knowledge, 
displayed in Figure 2.4, is generally accepted. It explains how humans use the spatial 
information of their surroundings to create a mental map or representation of the world 
around them. Thorndyke explains how each level is not self-inclusive, but in fact builds 


upon the knowledge of previous levels [THOR 80]. 
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Figure 2.4. Navigation Knowledge 


C. VIRTUAL ENVIRONMENTS 
1. Definitions of VE 


In order to discuss the effectiveness of VEs it is important to have an 
understanding of several of the major component definitions. 

Any model which provides frame rates of eight to ten frames per second (fps) for 
static environments and 60 fps for more dynamic environments can be said to have real- 
time graphics [DURL 95]. Real-time graphics over a network also must have a minimum 
of network latency (less than 0.1 seconds). 

Fidelity is more a qualitative than a quantitative classification. No defined scales or 
agreed upon distinctions exist. Waller et al. described environmental fidelity as the degree 
to which the variables in the model matched those in the real world [WALL 98]. Goerger 
defined high fidelity as “a model that represents lines of sight and terrain masking, 
provides realistic depictions of the vegetation and structures, and can provide a real-time 


interactive environment for the user" [GOER 98b]. Past studies have shown that 


increasing the fidelity of a model leads to an improved transfer to the real world [HAYS 
89][CAIR 96], but these results do not concern outdoor environments in particular. For 
proving the validity of the transfer of spatial knowledge, Goerger wanted to use as 
realistic looking a model as possible [GOER 98b]. Examples of the model used in this 


research and the corresponding real world photo are shown in figures 2.5 and 2.6. 





Figure 2.5. Model Photo Figure 2.6. Real World Photo 


Goerger described landmark models as “virtual representations of real world 
objects or locations that are easily identified, with defining characteristics, and are used by 
the participants as cues to navigate through the model” [GOER 98b]. These are in effect 
key landmarks, which must be included in any model built of a particular environment if 
that model is to allow the acquisition of landmark knowledge based on Thorndyke’s 
theory. Failure to include these essential features could result in the model’s 
ineffectiveness in representing an area and potentially inhibit the acquisition of landmark 


knowledge, which in turn may inhibit route and survey knowledge development. 
2. Prior Studies of Spatial Knowledge and Virtual Environments 


Chase examined the differences between individuals who had significant exposure 
to an environment, but had never seen a map or exocentric view of it, versus those who 
had only seen a map but had no direct exposure to "m environment [CHAS 83]. In the 
study, those with no direct exposure tended to have better survey knowledge, while those 
with only direct exposure tended to display better landmark and route knowledge. Chase 
reported that the repeated exposure to an area develops landmark and route knowledge, 
but that this exposure does not automatically develop survey knowledge. The results 
showed that use of a map provided the best survey knowledge but that the repeated 


exposure group outperformed the map group in landmark and route knowledge 
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development. Chase’s findings support how Thorndyke believed spatial knowledge to be 
acquired (see Section B. Spatial Knowledge Acquisition). The use of a map to rapidly 
acquire survey knowledge of an environment without prior exposure to it is also reported 
by Hirtle and Hudson [HIRT91]. 

Bliss, et al. examined the use of VEs in acquiring spatial knowledge [BLIS 97]. 
They wanted to see if exposure to a VE was the same as exposure to the real world. They 
broke up 30 firefighter subjects into three groups and experimented performing simulated 
rescue Operations in an office building. One group was map-only and was given a map to 
study first. A second group could first explore the building with a VE. The final group (the 
control) was given no training at all. Both the Map and VE groups clearly performed 
better than the control group in the results. Though the experimenters had tests and 
performance measure for landmark and route knowledge, they had no corresponding test 
for survey knowledge. Nevertheless, they concluded from their results that landmark, 
route, and survey knowledge could be gained from VEs. Their research did not show any 
advantage for the VE over the map. 

Waller, Hunt, and Knapp conducted a series of studies on the transfer of spatial 
knowledge in virtual environment testing [WALL98]. They devised a test involving the 
exploration of a real world maze. Subjects were divided into six different groups. The 
blind group had no exposure prior to the test. The real world group had one minute to 
explore while the map-only group had one minute to study a map. Three different VE 
groups were used: a desktop based VE, an immersive VE using a head mounted display, 
and a long-term immersive VE. Both the desktop and immersive VE groups were given 
two minutes to study. The long-immersive group was given five. The results indicated that 
the low fidelity VE system used did allow subjects to “develop useful representations of a 
large scale navigable space" [WALL 98]. The study found tbat with only short exposure, 
VE training was not more effective than map training. However, after significant 
exposure, the long-term VE group did outperform the map group. 

On the subject of the effect of immersion and presence on training, the results are 
mixed. Ruddle, Randal, Payne, and Jones showed no significant differences between the 


performance of immersive and non-immersive VEs groups in exploring virtual buildings 


[RUDD 96]. Wickens and Prevett showed that aviators who had an immersive viewpoint 
had better navigational performance over those who were given external non-immersive 
views [WICK 95]. This is possibly due to the more natural representation or intuitive 
viewer interpretation based on the field of view [WICK 98]. Other studies have shown 
that users estimate distance much more accurately if they are in an immersive VE where 
they physically rotate than if in one where they merely imagine they rotate [PRES 94]. 

Williams, et al. investigated the area of active versus passive control during flight 
mission preparation and its effect on performance [WILL 95]. The experiment looked at 
those subjects who actively controlled their flight of the aircraft in the VE versus those 
who passively watched a flight. The results indicated that those subjects actively involved 
performed better than those who merely observed. The researchers concluded from their 
data that the optimal VE designed to acquire spatial awareness would include active 
control by the user and not merely passive playback. 

Witmer's study into the transfer of spatial knowledge turned up useful information 
regarding the importance of the user interface [WITM 95]. He examined 64 subjects in 
navigating a large building. He divided his subjects into three groups: a real world group, a 
VE group, and a verbal directions only group. His results showed the VE group 
performing poorer than the real world group, but ahead of the verbal directions group. 
The results also indicated that spatial skills learned in a VE could transfer to the real world 
in some cases. Some VE subjects had their training seriously impeded by difficulties with 
the interface. These same individuals had difficulty performing the navigation task in the 
actual building, lending to speculation that poor interfaces can diminish the training 
effectiveness of a VE [WITR 95]. 

Darken and Sibert have conducted several experiments on wayfinding and virtual 
environments. In “Navigating Large Virtual Spaces” [DARK 96b] they concluded that in 
exploring a large VE without aids, subjects quickly became lost and disoriented. They 
found that inclusion in the model of a map, compass, or grid helped overcome this 
difficulty. They also found that the map provided superior navigational performance while 
the grid provided superior directional performance [DARK 96b]. In “Wayfinding 
Strategies and Behaviors in Large Virtual Worlds” [DARK 96a] they determined that 


individuals provided with supplementary aids in the VE, such as maps, were able to more 
quickly regain their starting position and maintain their orientation than when no aids were 
provided. They also noted that when exploring a VE, subjects tended not to revisit places 
they had already been. 

In the first iteration of this thesis experiment, Banker theorized that a combination 
of map study and VE exposure would provide the optimal solution for providing total 
spatial knowledge of an environment. Using a non-real-time model, he concluded that 
subjects of intermediate ability could successfully gain and transfer spatial knowledge from 
a VE to a real-world outdoor environment [BANK 97]. Banker’s study had three different 
groups: map-only, real world and VE. All of the subjects were active in orienteering 
though of different ability levels. None of the subjects was familiar with the specific testing 
area though many had been in similar terrain. Subjects were given one hour of study with 
which to plan and memorize a route through the environment. All of the subjects had 
exposure to the map during their study phase. After study, subjects were taken to the 
course and had to run their route from memory. Banker's results showed that ability level 
had a greater impact on the results than method of training, with advanced level subjects 
doing best overall. He noted a significant increase in performance for those classified as 
intermediate level orienteerers. Little increase in performance was noted in the advanced 
level or beginning level. Banker theorized that the ability and experience of advanced 
orienteerers allowed them to gain significant information from the map, which was not 
greatly augmented by the VE. He also concluded that the beginners were overwhelmed 
with information and too inexperienced to correctly focus on which information to study 
for success. 

Goerger, et al. conducted two studies similar to the Banker experiment. The first 
experiment used a complex man made environment to compare two groups: a map-only 
study group and VE study group. The study followed a similar pattern to the Banker 
experiment [GOER 98a]. Thirty minutes were given to both groups to study floor plans of 
the seven-story structure. The VE group also had a high fidelity real-time computer 
representation of the building, which they could explore during the thirty-minute study 


phase. The results of the experiment showed the map-only group significantly 
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outperformed the VE group. Goerger theorized that the short exposure limited the 
performance of the VE group. He concluded that “performance on spatial knowledge 
tasks after brief exposure to a high fidelity, real-time VE does not always exceed results 
gained from traditional navigation training techniques” [GOER 98a]. This theory is backed 
up by the results of Waller, Hunt, and Knapp. They found “short periods of VE training 
were no more effective than map training; however, with sufficient exposure to the virtual 
training environment, VE training eventually supersedes the map” [WALL 98]. Ruddle, et 
al. also found users performed better the longer they used the system and the more familiar 
they were with it [RUDD 98]. 

Goerger’s second experiment mirrored the Banker experiment in all details except 
that the model was a real-time model that allowed total freedom of movement, and that 
the subjects were not all active in orienteering. Like Banker’s experiments there were 
three groups: map-only, real world, and VE. The experiment used the same map and 
training area as the Banker experiment. After the individual's one hour of training, they 
were tested the same as in Banker’s experiment. Goerger concluded from his results that 
the spatial ability of an individual plays a "significant role" in the individual's performance. 
He also concluded that training conditions showed no significant effect on the ability to 
"obtain and demonstrate spatial knowledge of a natural environment" though the map 
group did perform better based on observation. Finally, he concluded that the 1:5,000 
scale map in the study was so effective and useful that it was hard to beat [GOER 98b]. 

The above studies allow us to conclude at least some benefit to using VEs for the 
acquisition of route and landmark knowledge. Map's, or other top-down representations, 
still seem to be the best at providing survey knowledge. Ideally, a VE would combine 
these advantages, allowing the user to either quickly use an external map or have one 


incorporated in the model. 


3. Model Classifications 


It quickly becomes apparent, given the vast scope of fields and tasks that are being 
explored with VEs, that no one set of standards fits all models. Much as with any real 


world building, different models must be designed for their express purpose and have 
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different features depending on that purpose. We would expect a VE for flight training not 
to expend great effort on modeling vegetation appearance since it is not the most crucial 
part of that task. In contrast, a VE designed for dismounted soldiers has to pay more 
attention to vegetation, but less to dynamic physics of falling objects. Goerger used the 
term “complex natural environment” [GOER 98b] to describe the areas of terrain complex 
enough to require detailed representation to portray them. While such a system may not be 


needed in flat desert or plains areas, it becomes more desirable in other terrain. 
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Figure 2.7. Fidelity vs. Movement Method 


Most VEs can be placed into one of four broadly defined categories based on 
terrain requirements. These levels are Fixed Wing Aircraft, Rotary Wing Aircraft, Ground 
Vehicles, and Dismounted Movement [SULL 98]. While there may be overlap between 


the fidelity and detail of terrain representation between the four levels, Fixed Wing 
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Aircraft is generally the least detailed with Dismounted VEs being the most detailed 
(Figure 2.7). 


Knowing the principal movement target of the model can help the model designers 
limit their work to the correct scope, and not add unnecessary details or leave out critical 
information. The model used for this thesis would fall into the dismounted movement 


category [GOER 98b]. 


D. LAND NAVIGATION AND ORIENTEERING 
1. Military Land Navigation 


Navigation is “the theory and practice of navigating” which means in turn, “to 
make one’s way” or “to follow a planned course” [WEBS 88]. Good navigation requires 
knowing the current location, the destination, the direction of travel (orientation), and a 
means of travel [WICK 92]. This closely matches the military definition of route planning 
which is “knowing where you are, where you are going, and how you are going to get 
there” [ARNG 83]. Navigation plays a critical role in the military, and as such an entire 
manual devoted solely to the training of map reading and navigation skills is maintained 
(FM21-26 Military Land Navigation). 

Army doctrine regarding land navigation says it is trained in a building block 
approach. First, basic map reading skills are trained, followed by dead reckoning. The 
more efficient, but more difficult to learn, terrain association follows. Later, soldiers 
develop route selection skills and techniques involving unit placement and planning tactical 
movements [FM21 93]. The military focuses on two basic techniques for navigation: dead 
reckoning and terrain association. Both have advantages and disadvantages. 

Dead reckoning involves first determining where a soldier is (based on their map- 
reading skills and compass readings), and then determining the direction and distance they 
need to go. Soldiers who are dead reckoning do not deviate from their path unless they 
come upon an uncrossable obstacle. In that case, they “box” around the obstacle and 
quickly return to their original route. The essential skills to the dead reckoning method are 


reading a compass and keeping pace count (for determining distance). While this method 
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is easy to learn, it is difficult to implement. Soldiers need to pay particular attention to the 
compass or they may veer. This is increasingly difficult on long movements where small 
errors in the beginning can have a dramatic effect at the end. Dead reckoning is also very 
unforgiving. Without any intermediate terrain association or landmarks, an error is only 
discovered once the prescribed distance is reached and the soldier is not in the right place. 
Dead reckoning can be useful for short movements, in dense terrain (jungle/forest), in 
featureless areas (desert/plain), or at night when other terrain features are not clearly 
visible. It can also be highly accurate over short distances. It has the advantage of being 
the easiest to learn and teach. Because of these characteristics, the Army advises soldiers 
to dead reckon a series of short distances between “known” points and locations [FM21 
93] to reduce the chance of error. 

The military prefers terrain association because it is more forgiving and less time 
consuming [FM21 93]. The essential aspect of terrain association is the ability to adjust 
the route on the way based upon confirmation of expected terrain features or landmarks. 
The steps in terrain association consist of first determining current location, and the 
desired destination. The map is examined for unique or distinctive feature of the terrain 
that can be used to guide the movement. Soldier’s plan their route to take advantage of 
these features to provide feedback. Terrain association allows them to take advantage of 
handrails, attack points, expanded objectives, checkpoints, catching features, and even 
vegetation [FM21 93][ARNG 83]. See Appendix P for definitions of these terms. To 
properly terrain associate, individuals should make sure they keep the map oriented 
properly, and they should refer to it often. The obvious disadvantage of terrain association 
is that the individual must be able to interpret the map and translate the world around them 
into topographic terms [FM21 93]. The ability to estimate distance and direction to a 
terrain feature, and quickly estimate position accurately are difficult to teach, learn, and 
most important, retain. Often these skills are only developed after long hours of practice. 

Often it is useful to combine techniques during navigation. The soldier may first 
terrain associate to an attack point (see Appendix P for definition) such as an intersection 


or bridge close to their desired objective, and then dead reckon from there. Army doctrine 
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recommends not becoming dependent on any one method but being capable of all [FM21 
93]. 

The recent emergence of Global Positioning Systems (GPS) technology has not 
diminished the Army requirements for land navigation training. While GPS can prove 
useful for determining location, the Army still requires that soldiers learn traditional 
methods. The seriousness that the military places on these skills can be viewed in the 
requirement for successfully completing the land navigation course prior to graduation of 
any of the basic military schools for officers and soldiers alike. Indeed, many of the more 
advanced schools have land navigation tasks that are essential to pass and are considered 
among the most challenging requirements (Ranger School and the Special Forces 


Assessment Selection course respectively). 
2. Sport Orienteering 


Orienteering is competitive land navigation. The sport of orienteering was first 
developed in Sweden in the 19" century and was brought to America in 1946 [FM21 93]. 
The International Specification for Orienteering Maps (ISOM) describes orienteering as a 
sport where runners complete a course of control points (or controls) in the shortest time 
aided by map and compass [ISOM 90]. In fact, there are four common types of 


orienteering events. 


The most common, and the one that most people think of as orienteering, is cross- 
country orienteering. Here, numerous points are spread out along the area. Contestants 
must visit the points in order but are not constrained as to which route they use to get 
there. Usually staggered start times are used. The runner with the best time who reaches 


all the controls in order wins. 


A variation of cross-country is score orienteering. Here the control points are 
again spread out but runners do not have to visit them in any order. Each control is given 
a numeric point value with the farther controls worth more. The runner who has collected 


the most “points” after a set time limit is judged the winner. 


Route orienteering is more of a training session. Here a leader will lead the group 


along a route. The participants have to draw the route followed on their maps and 
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annotate the locations of all the controls they saw. At the end of the route, all the maps are 


compared to the accurate master map. The closest map to the master map wins. 


Line orienteering is similar to route except that there is no leader. The participant 
follows a route drawn on his map to the best of his ability and annotates the locations of 


all controls along the route. 


For the speed events (cross-country and score orienteering) runners are usually 
provided with a clue sheet to assist them. This sheet provides data about the terrain 
feature the point is located in/on. The controls themselves are usually three-sided to be 


recognized from any angle of approach. 


Competition orienteering has different levels of competition. These color-coded 
levels increase the distance and difficulty of the locations. White is the easiest, followed by 
orange and then red. It is common for multiple courses to be run over the same area with 
different controls for the different courses littering the same area. Unlike military land 
navigation, which recommends at least 300 meters between points, orienteering has no 
such restriction. In any case, no two controls should be within 25 meters of each other and 
no two controls on the same course should be closer than 75 meters unless they are on 


different terrain features [FM21 93]. 


Another difference between most military and orienteering courses is the map 
itself. Most military navigation is done at 1:50,000 [FM21 93] or 1:25,000 at best. In 
FM21-26, the Army considers 1:50,000 to be reasonable for orienteering. Competition 
orienteering is usually done at 1:15,000 scale. This is a profound difference. Military 
navigators are looking to recognize major terrain features. Sport orienteerers are expected 
to distinguish between minute differences and the potentially close proximity of the points 


adds to the difficulty of this competitive sport. 
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III. METHODOLOGY 


A. EXPERIMENT OVERVIEW 

In order to determine whether Virtual Environments are useful for transferring 
spatial awareness of natural regions, a series of experiments was conducted. This section 
will provide an overview of the conduct of the experiments, while succeeding sections 
describe the tools, techniques, and methodology in more detail. The general sequence for 
the experiment was the in-briefing, battery tests, spatial orientation test, model 
familiarization (VE only), training in-brief, training, pre-course test (only for subjects in 
experiment #1), actual course test, and then debriefing. 

Upon the subject’s arrival, they were read the basic in-brief by the research 
monitor and had to fill out the initial consent forms. The in-brief is shown in Appendix C. 
The consent forms, including medical waiver are in Appendix D. 

Once the subject had filled out the consent forms, the next step was the battery of 
tests. Subjects had to complete a test on basic terrain identification, as well as a standard 
medical color-blindness test. Subjects completed a short informational questionnaire that 
asked where they first learned to navigate, how long they had been navigating, and at what 
level they would classify their ability. On a separate page, subjects had to complete a bar- 
line evaluation of their navigational ability. The redundancy of having the same 
information in two places provided a confirmation and removed any bias that may have 
been associated with the word descriptions of navigational level. Lastly, in this phase, 
subjects completed the Santa Barbara sense of direction scale questionnaire (see Appendix 
E). This 15-question questionnaire, developed by the University of California at Santa 
Barbara, asks the user to rate themselves on such topics as “I remember directions well" 
or “I always know where I am". This test helps assess how confident the subject is on 
navigational tasks. The map test, the navigation history and bar-line are in Appendix E. 

The next stage of the experiment was giving each subject the Guilford Zimmerman 
(GZ) spatial aptitude test [GOER 98b]. This test would determine a subject's ability to 
sense changes in direction and orientation. It was a 60-question test, with ten minutes 
allowed for completion. Subjects had to answer as many questions as they could, but 1/4 


point was deducted for each wrong answer from the total they answered correctly. No 
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points were deducted for unanswered questions. The GZ test cover sheet with an example 
question is shown in Appendix E. The test was used to segregate the subjects into high 
and low ability groups. Because past experiments proved a correlation between spatial 
ability and navigational performance, the goal was to equally distribute the high and low 
ability individuals into the map and VE groups. 

Once subjects were tested, their group was designated based upon GZ ability. 
Those chosen for the VE groups were now given 15 minutes of model interface 
orientation. For this training, they were put in an environment that did not resemble the 
actual model area. This training VE did have the same interface controls and allowed the 
user to become familiar with the various functions and capabilities available to them 
through the model. Prior to their completion of the interface orientation, they had to 
complete an interface familiarization checklist or, failing that, they would be required to 
conduct more training at the controls. During the course of this familiarization period, the 
monitor would note if the subject was having any noticeable difficulties with the model. 

Now all subjects were ready for their training phase. First, they were read a series 
of standardized instructions based upon their groups (Map or VE) and their experiment 
(Train-to-standard or Map Fidelity). In all cases, subjects were given a participant task list 
and important information on marking their map (see Appendix G and H respectively). 
This clearly laid out what the subject was supposed to do when they arrived at the testing 
site. This information helped focus the subjects during their training on what task they 
were preparing for. For the train-to-standard group, they were also informed of the pre- 
course test (see Section N.1. Train-to-standard) and the requirement that they pass this at 
the end of their training before they would be taken out to the actual course. The in-briefs 
for all four of the experiment groups are located in Appendix C. At this point the monitor 
would present the subjects their training aids, explaining each in detail. For the train-to- 
standard experiment group, these aids included a 1:5,000 orienteering map, an 
orienteering clue sheet, and side by side photo comparisons between the control points in 
the real world and in the model. The map-only group photographs only showed the actual 
points and not those of the model. For the map fidelity experiment, the 1:5,000 


orienteering map was excluded. In its place was an overhead photograph of the area, as 
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well as a 1:24,000 map of the region. To allow for easier reading, this map was also 
provided at 1:5,000 scale but with no additional information being provided on the map. 
These training aids are discussed in more detail in the following sections. 

Individuals would then actually conduct their training based on the standards of 
the experiment they were participating in. At the end of the training period, all subjects 
were to have their proposed route clearly marked on the map, showing the exact route 
they intended to follow on the actual course. During the training period, the monitor 
remained close at hand, recording the time they were taking, and answering any questions 
that the subject needed clarified. For VE subjects, monitors also noted any interface 
difficulties and if the subject appeared to be disoriented or lost in the VE. Upon the 
successful completion of the training period (which may have been an hour or more), all of 
the training materials, except for the clue sheet, were taken from the subject. The monitor 
and subject then left the training room and drove to Fort Ord. Once they neared the road 
leading to the test area, subjects were blindfolded to ensure that they would not arrive at 
the testing area already oriented. This was also done so that they could not see any of the 
control points in route (two points were directly visible from the boundary road under the 
right conditions). 

Once they were at the course location, subjects were again shown the target 
control point markers to remind them of what they were looking for. Subjects donned the 
GPS backpack that would record all of their movements on the course so that any error 
deviations could be accurately recorded. Subjects were read the Navigation Course task 
briefing, which outlined the exact task conditions and standards they should follow. See 
Appendix C for the briefing. The monitor initially oriented the subjects using the map and 
compass. The monitor also ensured that the subject had their clue sheet. The monitor 
encouraged the subject to speak aloud, so that the monitor could understand and record 
what the subject was thinking. The subject would have one hour to complete all the tasks. 
The monitor kept the official time. At control points two and four, the monitor would 
stop and have the subject answer questions. Time was stopped during these periods. The 
subject could also request the compass, the map, or the map and compass together at any 


time. All of these checks were timed and recorded. As the subject traversed the course, 
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the monitor recorded their comments, and noted when and where they made errors. As 
subjects completed a task, the monitor would read their next task from the Navigational 
briefing to ensure uniformity across the group. Subjects would have only one active task 
at any time. If subjects desired to change their route, they could do so. Once a subject 
found the control they were looking for, the monitor would read the next set of 
instructions. If the subject reached control point nine in the one hour time limit they were 
given the unplanned route task (see Section L). Once that task was complete, or upon 
being timed out at any portion of the test, subjects were guided back to the vehicle by the 
monitor where they performed the whiteboard test (see Section K. Whiteboard) regardless 
of whether they actually finished the course or not. 

Once the testing was finished, subjects were driven back to NPS. There they were 
given a detailed questionnaire about the experiment and the training tools they had been 
given. Subjects were also shown the route they had traveled by downloading the GPS data 
into ARCVIEW and presenting the data plots. Subjects were asked to elaborate on their 
thoughts at the times they made errors or mistakes during the course. Subjects were also 
asked the debriefing questions. These questions are shown in Appendix E. Once this was 
all finished, subjects were thanked for their participation and reminded not to talk of the 
experiment specifics to anyone. In total, the experiment took approximately four hours to 


complete. This could be much longer for the train-to-standard experiment. 


B. TEST ENVIRONMENT 

The area used for the experiment was a 1200 meter by 700 meter section of land 
located on what used to be Fort Ord in the coastal area of central California. The terrain in 
that region is characterized by shallow rolling hills and grasslands. The area used for the 
course varied from 90 meters to 123 meters in elevation providing for distinct navigational 
terrain features yet not being so extreme as to impose a physical challenge component to 
the experiment. The Fort Ord region of California has three predominant vegetation types, 
all of which can be found on the course (see Appendix F) 

Oak Forest covers much of the area (40%). Inland and coast live oaks of 25 to 45 


feet in height are dominant. These trees have large canopies that often touch their 
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neighbor, which can inhibit undergrowth and allow for easy traversal though poorer 
lighting. Some tree canopies can extend almost to the ground providing a visual as well as 
physical impediment. Where the trees are more scattered there is often thick undergrowth 
that can slow travel. 

Another 40 percent of the course is covered by maritime chaparral, a plant that 
grows in dense thickets that provides an extreme impediment to cross country travel. 
Movement in chaparral is almost always limited to the roads and trails in the area unless 
individuals plan to painfully “fight” their way through the dense brush. It is possible to 
traverse through Chaparral in some places but the rate of travel is very slow and the 
density limits visibility. Most travelers prefer to avoid these regions entirely or stick to 
roads. No test subject attempted to go through a section of chaparral more than once. 
Depending on the height of the chaparral, visibility ranges from excellent to poor. Often 
travelling on a trail through dense tall chaparral is the same as walking a corridor. 

The third category is perennial grassland. It covers the remaining 20 percent of the 
course and has knee high grasses with a scattering of trees and patches of dense 
underbrush. Excellent visibility and mobility characterize the region. 

The testing area is crisscrossed by a series of dirt roads, trails, and footpaths. Some 
date back to when the army trained in the area, others have come about due to public 
mountain biking and other outdoor recreation. The condition of these trails varies as they 
wind through the testing area, and it is not uncommon for a trail to be well beaten at one 
point only to be overgrown at another. The area also has numerous decaying outhouses, 
shacks, wooden pavilions, telephone poles and other manmade structures that scattered 
throughout the course, but usually located along roads or trails. The number of these 
structures and trails provides ample opportunity for parallel errors (see Section H. 
ERRORS). The area is also dotted with old foxholes, trench lines, pits, and cement pads. 
These features can aid navigation but can also induce parallel errors. 

Roads bound the testing area on all four sides. The northern road is paved but the 
other three are dirt. Many of the trails inside the course connect with one or more of the 
boundary roads. Travel into the area is prohibited to vehicular traffic without special 


approval. 
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C. MODEL 

The model is a real time replica of the test environment and was developed by 
MAJ Simon Goerger [GOER 98b] based on the aerial photograph, the Banker course 
map, and ground reconnaissance. Goerger built the model using the OpenGL language, 
EasyScene software, Coryphaeus, Inc. The model itself runs on a Silicon Graphics 
Industry (SGI) Onyx Infinite Reality workstation. This machine has four processors 
running at 194 MHz, 256 Mbytes of main memory, and 4 Mbytes of texture memory 
[GOER 98b]. Goerger studied extensively the required accuracy and developed his model 
at one meter resolution [SULL 98] [GOER 987b]. In order to have the model run in real 
time, Goerger made extensive use of level of detail (LOD) modeling. This meant that if 
some objects were far away they were not drawn, or were drawn in a simpler manner to 
save polygons. Goerger discusses at length the thought process he went through in 
deciding upon where exactly his transition points would be [GOER 98b]. 

To impart realism in the model, Goerger used digital photographs of trees and 
bushes attached to billboards to populate the vegetation in his model. Four different trees, 
three types of bushes, and two types of underbrush were used [GOER 98b]. These 
vegetation billboards were not placed to exactly match the individual trees found in the 
real world, but rather in a representational manner. Areas of dense vegetation in the real 
world would subsequently have more billboards in the model. 

In order to accurately place the roads, as well as give a sense of shadowing, 
Goerger overlaid the terrain with the 1993 aerial photo. He then hand placed the roads in 
their appropriate locations. All trails in the model are displayed the same, regardless of 
their real-world condition. This can lead to the “yellow-brick road” effect where subjects 
later doubted they were on the right trail because in the model it was clear, and in the real 
world it was overgrown. 

For the man-made objects, Goerger used digital photos and hand taken 
measurements to accurately build and locate the structures into his model. All told, the 
model included five outhouses, two shacks, two pavilions, nine cement pads, and three 


rock piles. It also had over 20 individual telephone poles, telephone lines, two trench lines, 
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and 200 sandbags. Over 9,000 meters of trails/roads were included, as were over 14,000 
tree billboards [GOER 98b]. Two military HMMWVs were added in an attempt to offer a 
sense of scale to the model. 

The interface uses a flybox (see Figure 3.1) as its interaction device. Users were 
allowed to independently turn their head to their right or left, or look up and down 
without affecting direction of movement or orientation. The users could designate a run or 


a walk mode. 





Figure 3.1. Flybox Interface 


The model’s avatar was set to a standard height of five feet eight inches above the 
ground, but users could toggle a 15-meter pop up view that enabled them to get above the 
trees. Users could also toggle an overhead “you are here” style map that showed their 
exact location and direction of travel. The maximum speed allowed in the model was 
roughly 10 miles per hour. Other features in the model were the ability to instantly teleport 
to the start or end points, as well as to any of the control points. Users could also get a 
quick view of any of the control points without actually moving there. Goerger designed 
the model so that the lighting could be adjusted to any one of six levels (night to bright 
daylight)[GOER 98]. Goerger did not implement any collision detection so users could 
unrealistically move through buildings and trees. Other than the 15-meter pop-up view, 
there was no way for users to raise or lower the avatar’s eye height. This created some 
difficulties in locating controls in the model, as subjects could not lower their head to see 


under tree branches. 
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To aid in navigation and to help the user orient themselves in the model, Goerger 
included a standard compass. This compass could be used during rotation, but could not 
be used while moving. If the user toggled the compass while moving, he would stop. The 
necessity of the compass in the VE was evident given the difficulty of ensuring accurate 
body movements in the VE (e.g., like turning around). 

For the visual display, Goerger decided upon a three-screen configuration with 
103-degree field of view [GOER 98b]. These three 40 inch screens were set up exactly 67 
inches from the subject to ensure the 103 degrees was achieved. He felt that this would 
give the user the best feeling of immersion and enable them to use their peripheral vision 
to maximum effect. Without this expanded viewing area, users of the model would be 
much more likely to make parallel errors or be forced to constantly turn their head. Figure 


3.2 shows the three-screen configuration. 





Figure 3.2. Three-Screen Configuration 


In summary, based on his extensive research into FOV, LODs, and his detailed 
measurements, Goerger was able to make an extremely accurate real-time model of the 
Fort Ord testing area. This allowed the user to seamlessly explore the region both on foot 


and in the air to quickly traverse to areas of interest. 
D. MAPS 


The experiments used two different maps. MAJ William Banker [BANK 97] 


developed the map used in the first experiment. This was the same map used in both the 
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Banker and the Goerger experiments [BANK 97] [GOER 98b]. This map was based оп а 
1993 aerial photo and was built to the ISOM [ISOM 90] specifications except it was set at 
1:5,000 scale. Note that traditional orienteering maps are 1:15,000, or in special 
circumstances 1:10,000 (see Section N.2. Training). The map clearly showed the many 
paths, trails, open areas, ditches, and even rock piles throughout the region. It 
differentiated between the vegetation giving seven different classification levels. The map 
also showed every man-made object, such as buildings, cement pads, and telephone poles. 
The map also indicated where telephone line was run. In sum, the map was incredibly 
detailed. Prior to their respective experiments, both Banker and Goerger confirmed the 
accuracy of the map [GOER 98b]. It was checked again in 1999 for these experiments. 
One deficiency in the 1:5,000 map was that the color-coding in orienteering maps is not 
the same as in traditional military maps. For instance, in orienteering, white space means 
“open runnable forest” and yellow means “open ground.” In traditional military maps, 
white means “open space.” This was initially confusing to some subjects. The other major 
problem with the Banker map was that the depression in the terrain that control four was 
located in was not clearly shown. This resulted in numerous subjects being confused and 
supposing four to be on a hilltop, not on low ground [GOER 98b]. 

The second map used was a commercially available 1:24,000 scale map of the Fort 
Ord/Seaside California region (see Appendix F). This map was purchased from the 
National Geological Department. Its basic terrain data was surveyed in 1929. It was last 
updated in 1983. On the flip side of the map-sheet was this same map blown up to 1:5,000 
scale. However, no additional information was added. The map-only had two vegetation 
classifications: open and forest. It showed only two trails running through the interior of 
the training area. No man-made objects other than the roads were annotated on the map. 
The map did not show any of the ditches, rock piles, telephone poles, concrete pads, or 
individual trees that were shown on the Banker map. The map did, however, clearly show 
the depression in the vicinity of control point four. 

The second experiment also used the 1993 aerial photograph mentioned earlier. 
This photo did not show any man made structures, but did show the interior trails and 


paths well. Some of these had changed since the time of the photo however (e.g., they had 
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become more pronounced, or become overgrown). The photo was close to 1:5,000 scale 
but was not an exact match, preventing the direct translation of distance from the map to 


the photo and back. 


E. TRAINING AIDS 


In addition to the maps and the VE, other training aids were provided to the 
subject during their train-up period. The most important training aid given was the 
orienteering clue sheet (Appendix F). The clue would be the only training aid that subjects 
would be allowed to carry with them out on the course. The clue sheet gave an overview 
of the course (total direct line distance between the points) as well as elevation changes 
between the points. The sheet also gave information about the feature or area near which . 
the control point was located. The purpose of the sheet was to jog the subject's memory 
about each control point so that they would not be trying to remember if they were 
looking for a pit, building, or a clearing. It was to aid the subjects’ memory. This sheet 
confirmed to the ISOM standard [ISOM 90]. 

The subject was also given photographs of the control points (Appendix I). If the 
subject was a VE participant, they were also given what the control point looked like in 
the VE as well. These photos were clear daylight pictures of the control points where the 
control points were clearly visibly. Note that the pictures were taken from the vantage 
giving the clearest view of the control, not necessarily the direction most subjects would 
approach the control from. If the VE pictures were included, they were taken from the 
same angle as the real world photo. These photos provided the subject with some idea as 
to the vegetation in the area of the control point, and could help fix in their minds the 
information provided by the clue sheet. | 

The last aid was the compass. This standard Silva compass was available for 
compass checks during the run of the course. The monitor ensured that all subjects were 


familiar with this device once they got to the course. 
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Е. DATA GATHERING TOOLS 


In order to measure whether subjects were following their planned path within the 
margin of error a differential global positioning system (DGPS) backpack was used. This 
system would take location information transmitted from satellite and record it in a 
Newton MessagePad 130 (see Figure 3.3). These data plots were taken every five 
seconds. Once the experiment was finished, the location plots were downloaded from the 
MessagePad into a PC. Using FieldWorker software, the data was overlaid onto an 
overhead photo of the region. This enabled the monitor to accurately measure exactly how 
far off route subjects had traveled during errors. This program was also invaluable during 
the out-briefing for showing the individual exactly what they had done, and for discussing 


exactly where they made mistakes. 





Figure 3.3. DGPS Backpack and MessagePad 130 [GOER 98b] 


A second means of data collection was the monitor himself. He would record the 
actions of the subject as they progressed through the testing environment. All map-checks 
and route changes were recorded, as was the time it took individuals to arrive at control 
points. The monitor also recorded the exact time that subjects left their planned route, 
where the error occurred, and the type of error. This was needed in case of DGPS failure. 
Subjects were encouraged to "think out loud" (Appendix K) and the monitor recorded 


their comments and actions. 
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G. COURSE 

The testing course itself was design and set-up by MAJ William Banker 
[BANK97]. He used the International Orienteering Federation’s specifications [ISOM 90] 
to develop the course and the course map (Appendix F). In all but one respect, the course 
conforms to thee standards of an orienteering “orange” course [ISOM 90]. The only 
deviation from the standard is the level of map detail. Orienteering competition normally 
uses a 1:15,000 scale map. The official specification allows the use of 1:10,000 in 
exceptional cases. MAJ Banker’s map (Appendix F) is of 1:5000 scale (see Section B. 
MAPS). The other map provided was a 1:24,000 scale map that was not detailed enough 
by sport orienteering standards. However, the military considers 1:50,000 to be the 
standard navigation map and they seldom see anything better than 1:25,000 [FM21 93]. 
The course greatly differs from standard military land navigation courses in that the 
control points are not at least 300 meters apart and the overall course is not over 2500 
meters in length. 

The start point for the course is the intersection of Gigling and Watkins roads on 
old Fort Ord. The course consists of the start point and nine control points which must be 
found in order. If a point is discovered out of order, no credit is given until the point is 
found again is the proper sequence. The points were placed so that there were numerous 
routes to the controls. No controls were located directly on a road or trail as per ISOM 
standards. Because of the layout of the course, it was not efficient to reuse previously 
traveled routes during the conduct of the course. Three points were located below ground 
level. Control point two was in a narrow pit, point four in a wide shallow depression, and 
control seven in a dry ditch. No points were hidden in dense brush or inside man-made 
structure. All controls were located near identifiable terrain features or objects. The first 
three controls in the course are located relatively close together. This often allows the 
subjects to gain a sense of confidence. Direct line distance between the points on the 
course is just over 2000 meters, though a route following only roads and trails could easily 


double this. 
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H. ERRORS 

There are five different classifications of errors that individuals commit during the 
course. Again, as noted earlier, individuals are assessed an error if they travel more than 
five meters off of their designated path (if path was planned on a trail/road) or if more than 
15 meters off of a cross-country route. Once an individual is assessed an error, the 
distance he travels is calculated until he returns to his preplanned route, marks a new 
route, finds the control point, or his one hour time limit for the experiment expires. 
Individuals can commit multiple errors if they move off their route, correct themselves, 
and move off their route again. In addition, if they mark a new route and then veer off that 
they also get multiple errors. Each of these errors was recorded and weighted differently 
as noted in the following discussion. 

By far the most common type of error encountered was a mirror error. This is 
where the subject, upon reaching a decision point (a branch in the trail for instance) 
chooses to follow the incorrect one. Given the numerous intersecting trails and roads in 
the course, it was quite common for individuals to make this mistake. Given the high 
potential for this type of error, it was given a basic weight. Figure 3.4 shows a mirror 


error. 


—p  - Original Direction of Travel 


= > -Correct Route 


E a -- Mirror Error 





Figure 3.4. Mirror Error Example 


Another common type of error is a parallel error. This occurs when the individual 
thinks he is at one location but is actually at another. The subject may be looking for a 
building next to an intersection and find a different location with the same features. The 


subjects mistakenly thinks he is at his desired location.This course has numerous similar 
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trail intersections, buildings, and locations that make this type of error common (see 


Section A. EXPERIMENT OVERVIEW). 





Figure 3.5. Parallel Error Example 


A compound error occurs when individuals make a mirror or parallel error off a 
newly planned route. This can occur after a reorientation error (see later), or if an 
individual himself requests a new route and then errs off it. 

An out-of-bounds error occurs when a subject leaves the course boundaries. While 
there is no restriction against using the boundary roads, should an individual take even one 
step off the wrong side of a boundary road he is assessed an error. The individual is not 
oriented as to where he is on the map if he makes this error. He is only informed that he 
has left the course. There is no error distance recorded with this type of error. Because 
being told you left the course boundary provides some information and since error 
distance is not calculated, this type of error is tracked under map-checks. 

Should a subject leave their planned route for more than 15 continuous minutes 
and not be making progress towards their next control point, they will be reoriented as per 
Navigation Course task briefing (Appendix C) and be assessed a reorientation error. At 
this time the subject will be shown their location on the map, and given one minute to plan 
a new route to their next control point. Since the subject uses the map and is shown their 
exact location, this type of error is recorded under map-checks and not distance errors. No 
error distance is recorded for this type of error since individuals may make compound 
errors later, which would result in the double counting of error distance. 

Note that when individuals made an error, only the initial cause of the error was 


recorded not any subsequent misjudgments they may make during the course of the error. 
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Thus if a person makes a parallel error and thinks he is at a different location, and he takes 
a wrong trail based on that, it will be a parallel error and not a mirror error. Also, if a 
subject, after taking a wrong trail (mirror error) then assumes they are at a different 
location (parallel error) and make decisions based on this erroneous conclusion, it is still 
recorded as a mirror error. Once an individual has made an error, error distance continues 
until: the subject returns to their planned route for the control point they are looking for 
(at any point along the route), the subject finds the control point, the subject creates a new 
route (though they may start a new error if they plan this new route not from their actual 


location) or the experiment’s one-hour time limit expires. 


I. SUBJECTS 

The subjects for this experiment consisted of 20 individuals ranging in age from 29 
to 40 with an average of 33. All were active duty Army (11) or Marine (9) officers of 
either Captain (17) or Major (3) rank. 75% of them had ten or more years of experience 
navigating, with the rest having five or more years. 95% of the subjects rated above 
average on the Santa-Barbara sense of direction scale. Most self-assessed their ability as 
intermediate, with several claiming expert. None felt they were beginners. However, when 
evaluating themselves using the bar-line, two subjects did place themselves in the beginner 
category. None scored lower than 17 out of 20 on the map reading test. Only one was 
color-blind. Two subjects were female. All the subjects had no prior knowledge of the 
testing area, nor had they any experience with the maps used or the virtual environment. 
There was no tangible compensation for the subjects. Data was collected from 22 April 
1999 to 23 July 1999. The subjects were divided into four groups: train-to-standard VE 
(VE1), train-to-standard map-only (MAP1), low-resolution-map VE (VE2), low- 
resolution-map map-only (MAP2). 


J. WHEEL TEST 
The wheel test was used twice during the conduct of the actual course as a means 
of determining if the subjects acquired survey knowledge. The tests were taken at control 


points two and four. At both points, subjects were asked to point towards one location 
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they had already visited and two they had not. In no instances were these locations the 
ones just previous or next in line. At control point two subjects point to the start point, 
control point five, and control point nine. At control point four they point to control point 
one, six, and eight. The time it takes the subject to complete this task is recorded. Also 
what direction they face is recorded. Once the subject finishes, the results are 
photographed and measured on the spot. During this procedure, the official clock is 
stopped so that the individual is under no time constraint pressure. The actual wheel 
consisted of a 12 x 12 sheet of laminated cardboard with a seven-inch full color wheel (see 
Figure 3.6) and three numbered arrows. The wheel had 16 colored segments. Colored 
segments were used so that the individual would not confuse the wheel with a compass 


[GOER 98b]. Individuals were allowed to orient anyway that they choose during the test. 





Figure 3.6. Wheel Test 


K. WHITEBOARD TEST 

At the completion of the course, all subjects are given the white board test 
regardless of whether the found all the control points or not. Like the wheel test above, 
this test evaluated the subject’s survey knowledge of the course area, but in a more 
exocentric manner. This test consists of a blank white board and ten colored chits 


representing the control points. Subjects are instructed to lay the chits out in spacing and 
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orientation to match the real world layout of the course. Figure 3.7 shows a subject doing 
the whiteboard test. This layout is photographed and then analyzed later to determine how 


far off from the real world it is. 





Figure 3.7. WhiteBoard Test 


L. UNPLANNED ROUTE TEST 

Upon completion of the course, subjects who successfully found all nine of the 
control points would have their survey knowledge tested again. They would be asked to 
point to control point four, and how far they thought they were from it. They would be 
asked to describe how they would get back to control point four if they had to go there 
again. After describing the route, subjects would be given ten minutes to follow the route 
they had described. The monitor would record if they followed the route that they 
described, what errors (if any) they made, and if they found the control point in the ten 
minute time limit. If the subject made an error, the cause of the error and the error 
distance would be recorded. This data would later be analyzed to see how well they knew 
their environment and could they conduct an unplanned movement without aid of map or 


compass. 


M. RESEARCH MONITOR 
The research monitor followed the text in the experimental outline to the letter to 
ensure uniformity of test experimental conditions to all subjects. During the train-up 


period, the monitor observed the subjects to see how they conducted their time. He was 
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on hand to answer specific questions about the course, record where individuals had 
difficulty during the train-up time, as well as administer the pre-course test. 

The purpose of the research monitor during the course was to record the 
comments and actions of the test subject (in addition to the tracking tools) while 
remaining in the background. This meant prompting the subject for their thoughts but not 
distracting them from their task. The monitor kept the official time, which he stopped only 
when recording information or asking the subject questions. The monitor also needed to 
record when errors, map checks, and route changes occurred, and to ask the in-course 
questions (see wheel test, whiteboard test, and unplanned route test). The monitor 
recorded the time the subject arrived at the various control points and had to be prepared 
to orient the individual as necessary (see re-orientation error). Finally, the monitor carried 


all the water and supplies other than the clue sheet, which was carried by the subjects. 


N. TRAINING 
1. Experiment # 1: Train-to-Standard 
In his conclusions, Goerger speculated that the limited training time may have had 
an effect on the outcome [GOER 98b]. Based on his data he felt VE subjects who actually 
performed multiple runs through the simulated environment did better, at least in terms of 
finishing the course. In addition to the lack of iterations, Goerger also felt that the short 
training time may not have allowed VE subjects to both explore the environment, practice 
potential routes, and build a valid mental map of the environment all in the span of one 
hour [GOER 98b]. He envisioned that as exposure time increases, the performance of 
both the VE and real-world groups would exceed that of the map-only group. Based upon 
these conjectures, the training portion of the experiment was ripe for variation. 
Educational studies involving VEs have shown that students learn best when 
“multiple practice” is included. In this case multiple practice includes not only “repeated 
practice of a topic" but also “practice from multiple representations” [MCLE 96]. 
The Army places heavy emphasis on training. It is how they prepare for war and 
the commonly repeated mantra is “Our top priority is training" [FM25 88]. Army 


doctrine demands that training be performance-oriented, hands-on, and challenging. Since 


training is sO important to the Army, it requires that all training events have tasks, 
conditions, and standards. The Army feels soldiers perform better when given these 
guidelines [FM25 90]. The tasks are the concepts to be trained. The conditions are the 
tools that can be used and the surrounding area. The standards are what the soldier must 
be able to do in order to be considered “trained.” 

FM25-100, Training the Force describes the Army’s view of training to time 

versus training to standard: 

Demand training standards are achieved. Leaders anticipate that all tasks 

will not be performed to standard. Therefore design time into training 

events to allow additional training on tasks not performed to standard. It is 

more important, however, that they achieve the established standard on a 

limited number of tasks during a training event than to attempt many and 

fail to achieve the standards on any [FM25 88]. 

The Army also says training becomes more effective when it is standard based than when 
it is time based or procedure based [FM25 90]. 

When designing the original experiment, Banker arbitrarily chose the one-hour 
training time. No analysis was done to determine if this was an optimal time, or even if it 
was sufficient. The Goerger experiment continued the use of this arbitrary time limit to 
training. Based upon Goerger’s suspicions, as well as the supporting research from other 
VE studies, and most importantly, from the Army philosophy towards training, the first 
experiment did away with the arbitrary time limit to training and instead incorporated a 
standard based system. 

Having decided to implement a standard based system, it was necessary to decide 
exactly what the standard should be. Because the experiment used two different group 
with two entirely different media (maps-only vs. VE) it was apparent that the standard for 
each group would be slightly different, yet had to be as similar as possible. The first step in 
this area was determining what exactly the experiment was testing. While the context of 
the experiment would be looking at survey knowledge at many places, the most important 
aspect to completing the course was developing the landmark and route knowledge. Thus, 


the experiment needed a standard that tested this learning. 
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For each group, a pre-course test was developed. When the subject was read their 
training instructions they would be informed of the pre-course test and told its standards 
from the start. They were informed of the requirement to pass this pre-course test in order 
to finish their training period. There would be no time limit to their training. When they 
felt they were ready, they would attempt the pre-course test. If they passed to standard 
then their training would end and they would proceed out to the course. If they did not 
pass the test to standard the pre-course test would be stopped and they would be told 
why/how they failed. They would then be given the ES to conduct more training 
before attempting the test again. There would be no limit on the number of times the 
person could potentially test. 

For the VE group, the design of the pre-course test was straightforward. The 
subject would be forced to run their selected route in the VE and actually prove that they 
could find all the points (at least in the VE). They had to be allowed access to their map 
during this training period, as map checks were available out on the course. However, the 
experiment didn’t want individuals to run the test with map in hand, so they were limited 
to two thirty-second map checks. As in the real experiment, they would not be allowed to 
move during a map check. During the pre-course test, the subjects were also not allowed 
to use the 15 meter-overhead view, the top-down view, nor the teleportation features of 
the VE. These, while potentially useful for training, would not be available in the actual 
course so were prohibited to ensure that the subjects were not reliant on them. Due to the 
difficulties of determining exact orientation in a VE, the use of the compass could not be 
prohibited during the pre-course test. However, subjects were informed that their usage 
was monitored and if it were found that they were overusing the compass, they would be 
failed. Overuse in this case was regarded as constantly stopping to check direction of 
movement. All compass checks during the pre-course test were recorded as well. 

As it was possible for people to get lost and recover in the actual course, this 
possibility was incorporated into the pre-course test. If subjects left their designated route 
(based on monitor observation) they were assessed an error. After five minutes if they had 
not corrected themselves, then they were stopped and oriented by the monitor. Subjects 


would be allowed to be oriented twice. If they required a third orientation they were 
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considered to have failed. In addition, if subjects left the course boundaries, they failed. 
Note that due to the inability to actually see through the branches of the trees in the 
model, individuals actively searching for a point in the correct vicinity were not assessed 
to be in error. The pre-course briefing for the VE group is shown in Appendix C. 

For the map-only group, devising the pre-course test was more difficult. Clearly, 
the subject had to show that they in fact had a clear understanding of where they wanted 
to go and how they would travel between the points, just as the VE group had to do. The 
map group, however, lacked the simulator with which to demonstrate this knowledge. The 
best way for the subjects to pass along this information was to verbally talk their way : 
through the route they would follow. Appendix C shows the instructions the Map group 
was given. As with the VE group, the map-only group was given two map checks so that 
they could refresh their memory of their route if necessary. The monitor encouraged the 
subjects to be as detailed as possible on how they would travel, but did not ever define a 
necessary Style for fear of biasing the results. For instance, if the instructions required the 
subject to mention distance, then that may have perhaps changed their planning. Thus no 
examples were given to the subjects; they were just encouraged to be as detailed as they 
could be. Thus, it was left to the monitor’s judgement if the subject had a clear picture in 
his mind of how he would travel between the points. If the subject was vague or described 
a route other than what they showed on their plan, the monitor would ask them to 
describe the route segment again, preferably in more detail. More than two errors of this 
type would result in failing the pre-course test and return to the study time. It is important 
to note that while the monitor would ask for clarification if the subject described a wrong 
turn, the monitor did not notify subjects if they had misidentified a terrain feature on the 
map (i.e., a road as a stream, or a depression as a hill). This was done so as not to remove 
on of the potential hazards of reading a map, the misidentification of a feature. This was 
felt to be fair, as it was also possible for the VE group not to correctly associate the 
terrain or to misidentify a landmark and still find the points [GOER 98b]. The test was to 
determine if the subject had a clear picture in his mind as to how they planned to conduct 


the test, not a guarantee that it was correct or would be successful. 
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2 Experiment # 2: Varying Map Fidelity 

For the second experiment, it was decided to go back a focus on the map given to 
the test subjects. As explained earlier in Section D. MAPS, Banker designed a 1:5,000 
scale orienteering map for the experiment which Goerger also used [BANK 97] [GOER 
98b]. Goerger had suspected that the fidelity of this map might have skewed the 
performance of the subjects in his experiment [GOER 98b]. He estimated that 
performance would decrease for a map-only group based on map scale. Figure 3.8 


visualizes Goerger’s supposition. 
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Figure 3.8. Goerger’s Performance vs. Map Resolution Theory 


Remember that the Army considers 1:50,000 to be the standard and sufficient scale 
for land navigation, with 1:25,000 being the absolute best available for training and 
operations [FM21 93]. Also, keep in mind that ISOM states that maps should be at 
1:15,000 scale for competition, with 1:10,000 scale maps being allowed in exceptional 
cases and requiring official approval [ISOM 90]. In all cases, Goerger’s subjects were 
military officers or civilians with orienteering experience. This group was either familiar 


with navigating given 1:15,000 orienteering maps or with 1:50,000/1:25,000 topographic 
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maps. The subjects, when presented with a greater detail of information and in a format 
they were used to, may have been able to gain an advantage based on the level of detail 
alone. This is based, in part, on the observation of real world and VE subjects using part 
of their time to explore possible routes, not merely memorizing the selected route. Map- 
only subjects seemed to quickly decide upon a route and then spend their time memorizing 
it [GOER 98b]. This can also be observed in the after action questionnaire results. The 
subjects rated the map as more than adequate with it averaging a 4 out of 5. 

In order to determine if the scale of the map had an impact on the results, the 
experiment went back to the methodology of the original Goerger experiment [GOER 
98b] (see Chapter II). While the same methodology was used, the map itself was changed. 
Instead of the 1:5,000 map created by Banker, subjects were given a more realistically 
scaled two-sided map sheet. On one side the map sheet was a to scale 1:24,000 map of the 
course taken from the commercially available 1:24,000 scale map of the Fort Ord/Seaside 
California region. This map was purchased from the National Geological Department. Its 
basic terrain data was surveyed in 1929. It was last updated in 1983. On the flip side of the 
map-sheet was this same map blown up to 1:5,000 scale. However, no additional 
information was added. In effect, it just made the map easier to read. While this showed 
the same basic terrain as the Banker map, there were some significant differences. First, 
the 1:24,000 map showed one kind of vegetation, compared to the seven different types 
on the Banker Map. In addition, the trail network shown on the new map was nowhere 
near as extensive as the Banker map. One thing the new map did make much clearer, 
however, was the depression surrounding control point four. This had been difficult to 
discern on the Banker map, but was readily visible on the new map. 

The second training aid added was the overhead photo of the region. This photo 
was used in the VE model where it provided shading in the top-down capability [GOER 
98b]. In order to ensure fairness, this information had to also be provided to the map 
group. This is not entirely unrealistic either. For most important missions, units will be 
provided with aerial photographs of the objective area, even if not recent. This photograph 
was taken circa 1993 and the area had had minor changes to the trail network from what 


was shown in the photo. While the control points were shown on the photo, no legend 
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was provided concerning vegetation or distance. While the photo was close to 1:5,000 
scale it was not an exact match, so subjects could not directly transfer distance 
measurements from the map to the photo and vice versa. Neither the new map nor the 
photo showed any of the mad-made structures that had been so clearly shown in the 
Banker map. They also did not show the breaks in the vegetation, the tree lines, nor the 
small depressions, pits, trenches, or rock piles. 

As mention earlier in this section the conduct of the experiment was the same as 
the Goerger experiment. The exact wording of the various experiment sections are found 
in Appendix C. In addition to the map changes, one additional change was made in the 
conduct of the experiment. Because of the inclusion of two different training aids for the 
map (the map and the photo), the time allowed for a map check and for the map and 
compass check together was increased. For this experiment map checks were given one 
minute, while map and compass checks together were given one minute and thirty 
seconds. This extra time was allocated to allow the individuals to peruse both training aids 
during any map checks. Other than the change in training aides and the difference in the 
time for the map checks, all other aspects of the training period and testing were the same 


as in both the Banker and Goerger experiments. 
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IV. ANALYSIS 

A. RESULTS 

1. General Information 

The two different experiments used in this thesis were designed to test the two 
hypotheses regarding the spatial knowledge gained by participants exposed to different 
training methods. Participant’s overall performance was evaluated based on the data 
acquired during their conduct of the actual orienteering course. This data comes from the 
results of the participant’s performance on the landmark, route, and survey tasks. 

a. First Experiment Hypothesis (Train-to-Standard): 

Given extended exposure to training materials and the requirement to meet 
a set standard before continuing, individuals with access to a real-time virtual environment 
will outperform those who are only exposed to a map and photos of the control points 
with a similar standard. 

b. Second Experiment Hypothesis (Map Fidelity): 

Given an hour exposure to training materials to include a 1:24,000 scale 
map of the region, individuals with access to the VE will perform less poorly compared to 
their counterparts with a 1:5,000 map than those of the map-only group to their 
counterparts with the same conditions. 

2 Power Analysis 

The results of the experiments are presented as box plots with the mean, standard 
deviation, and standard error shown. Often extreme outlying data is depicted as dots on 
these plots. Most of the tests conducted on the data were two way analysis of variances 
(ANOVA). The primary analysis was based on the training group, but a host of possible 
factors were analyzed and are discussed in Section B. The sample size of this experiment 
combined with the Goerger experiment [GOER 98b] is 35 subjects in seven different 
groups. An « value of 0.05 was used to determine significance. The resulting power value 
(1-B) was 0.3051. Due to the high number of degrees of freedom and relatively low 
sample size, it may be unwise to draw conclusions based solely on the failure to determine 
a positive effect. Note that as with the Goerger experiment [GOER 98b], the simultaneity 


of effects was not considered so only ANOVA tests were conducted. This is not to say 
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that there are no multiple factor effects, just that this was not examined as part of this 
thesis. Future work may wish to reexamine the data using multiple analyses of variance 
(MANOVA). 

Note that for the charts shown, the following symbology is used: MAP is used to 
represent those subjects who were map-only in the original Goerger experiment. MAP1 is 
those subjects who were map-only in the train-to-standard experiment. MAP2 is those 
subjects who were map-only in the low-fidelity map experiment. RW represents the real- 
world subjects from the Goerger experiment. VE represents the Goerger study VE 
participants. VE1 is those subjects who were in the VE group for the train-to-standard 
experiment. VE2 is the VE group for the low-fidelity map experiment. 

3. Normalization of Data 

Some of the participants in the experiment were unable to complete all of the tasks 
because of the one-hour time limit during execution of the actual course. This is because 
some of the measurements occur at various locations spaced out over time and space. To 
make the subject's data comparable, many of the measures were normalized over the 
number of controls attempted to put the data into a format for comparison analysis. This 
normalization technique was used in evaluating average number of errors, average error 
distance, average distance per error, and average mapcheck scores respectively. 

4. Landmark Knowledge 

A subject’s landmark knowledge was determined by their ability to locate the 
control points during the experiment. Subjects were given one point for each control they 
found (points had to be found in order). If the subject was timed out prior to completing 
the course, they could receive partial credit for the next point on their route. Subjects who 
had been heading to the next point and who were on their planned route received 2/3- 
point credit. Subjects who were off their planned route received 1/3 point credit. Subjects 
received no credit for points that they did not visit nor attempt. This scoring criterion is 
the same as in the original Goerger experiment [GOER 98b]. This score was combined 
into each subject's landmark score. 

The landmark scores for all seven training groups are shown in Figure 4.1. 


Statistically the means are not different to our &« standard of 0.05, F(6,35) = 2.395, P = 


0.054. However, statistical differences were noted between the VE and MAPI, VE and 
MAP2, and VE and VEI. Observation of the results suggests that the train-to-standard 
groups perform the best with the VEI train-to-standard group achieving a perfect result 
(all subjects finished the course finding all the controls). The map-only train-to-standard 
was not far behind with all subjects finding eight or more controls. This suggests that 
subjects who trained to a standard, and not just for an arbitrary time, learned the course 
better. Furthermore, those that had time to explore in the VE were able to gain the most 


landmark knowledge. It also lends credibility to the first hypothesis of this thesis. 
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Figure 4.1. Box Plot of Landmark Score (Group) 


The strong landmark performance of the MAP2 group (map-only group low- 
fidelity map) does not fit the experiment's hypothesis. The most likely explanation 
involves route simplification. This explanation is described in Section B. 

When evaluating the results strictly from the macro-group perspective, the analysis 
is far less distinct, with F(2,35) = 1.236, P= 0.304. We are thus not able to state any 
conclusive results from any given treatment source (RW, VE, or MAP). This lack of 
significance is not surprising given the very strong differences noted above between the 


various treatment groups. Figure 4.2 shows the box plot of the macro-group. 
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Figure 4.2. Box Plot of Landmark Score (Macro-group) 


Э. Route Knowledge 
For this thesis, a subject’s route knowledge was determined by evaluating how 
well they followed the path that they planned during their initial training period (see 
Chapter HI, Section A. Training Overview). Three different measures were judged to 
determine exactly how well the subjects kept to their route. 
a. Normalized Error per Control 
First, the number of errors committed, averaged per control point (and 
normalized as mentioned earlier) was compared. The types of errors were discussed in 
Chapter III Errors. For this analysis, only mirror, parallel, and compound errors were 
included. Reorientation errors and out-of-bounds errors were included in the map check 
results, to be discussed following. Individuals who made fewer errors per control point 
were considered to have performed better. 
Figure 4.3 shows the normalized average error score for all seven of the 
subject groups. All three of the map groups are about the same with between 0.6 and 0.7 
errors per control attempted. This is followed by the real world at 0.78. The biggest 
difference between the groups comes from the VE groups. The train-to-standard VE 
outperformed everyone with 0.40 errors per control. This was followed closely by the 
low-fidelity map VE group at 0.48 and the original VE group trailed everyone with 1.0 


errors per CP attempt. 
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Figure 4.3. Box Plot for Errors per Control] Attempted (Group) 


The means between groups are statistically different, F(6,35) = 2.497, P = 0.0462, 
with the significant differences coming between the VE and VEI group, VE and VE2, 
RW and VE1, VE and MAP1, and VE and MAP2. The analysis of the data seems to show 
a remarkable improvement for the VE subjects when trained to a standard as opposed to 
just a time limit, as in Goerger's experiment. Direct observation of the results suggests 
that the map-only subjects for the train-to-standard condition also showed improvement 
over the other map-only counterparts, but to a much lesser degree. Subjects who train-to- 
standard with maps had slightly lower average number of errors compared to the other 
map-only subjects (0.600 vs. 0.676[MAP] vs. 0.644[MAP2]). This result would tend to 
support Goerger's theory that the information that can be gained from a map will level off 
over time [GOER 98b]. These results seem to support the first hypothesis of this thesis; 
subjects who trained to standard with the VE not only found more controls than the other 
groups (see Section A.4. Landmark Knowledge) but made fewer errors than those who 
trained to time. This superior result would have been even lower had not one VE] subject 
(VE1-3) twice decided to go down follow a path he believed to be wrong, just to confirm 
that it was wrong! The results of the VE2 and MAP2 groups are surprising and 


unexpected. À possible explanation is found later in Section B.1. 


DS 


When the results are examined using the macro-groups (MAP, VE, and 
RW), no statistical difference between groups is found, F(2,35) = 0.411, P = 0.6662. 
However, direct observation shows the MAP macro-group with a much smaller standard 
deviation over the course of the experiments. This implies that regardless of how long they 
studied, or the level of information on the map, the map-only group performed about the 
same, at least as far as number of errors. One of the reasons for this is the familiarity of the 
military officers in this study with maps and map reading. Figure 4.4 shows the box plot of 


the macro-groups. 
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Figure 4.4. Box Plot of Errors per CP Attempt (Macro-Group) 


b. Distance Traveled per Error 

The number of errors a subject makes is just one factor of route 
knowledge. Just as important is determining how long it takes a subject to determine they 
have made an error and recover. While no exact time measurements were taken to 
determine how long this took, this was evaluated in terms of distance traveled from the 
point of the error until the individual returned to their planned route or found the correct 
control. It was possible for subjects to never recover from an error; either because time 
ran out, or because 15 minutes elapsed and the monitor reoriented them. For purposes of 
this test, all of a subject’s error distance for a point was added and then divided by the 


total number of errors for that point. Adding the subject’s totals for each control point and 
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then dividing by the number of controls attempted (to normalize the data) gives the 
average distance per error per CP attempted. Goerger called this the Total Error Score 


[GOER 98b]. Figure 4.5 shows the results of this split by training group. 
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Figure 4.5. Box Plot for Total Error Score per CP Attempt (Group) 


The results of the analysis of means is not statistically significant overall, 
F(6,35) = 2.188, P= 0.0743, but there were major differences between the VE1 and VE, 
VE2 and VE, MAPI and VE, MAP2 and VE, and VEI and RW. Direct observation of the 
data shows that subjects of the train-to-standard groups (VE1, MAP1) not only made 
fewer errors than other groups (as mention in the previous section), but also traveled less 
distance on those errors before they corrected themselves. This seems to show that these 
subjects had a better understanding of the route they planned to follow and where they 
were on that route at all times. It also means they recovered quickly if they did veer off 
course. The low-fidelity map groups also performed very well. The rationale for their 
improvement is explained in more detail in Section B.1. 

The results of the macro-groupings were also analyzed to see if any 
significance could be found from the overall training group. With a variance of means of 
F(2,35) = 0.422, P = 0.662, no significant effect can be concluded based on the macro 
grouping. Figure 4.6 shows the box plot of the distance per error based on the macro- 


groups. While no statistical conclusion can be reached, direct observation shows that the 
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map as a whole resulted in much less standard deviation among the various groups. Again, 


this can be attributed to the familiarity of the subjects with map reading. 
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Figure 4.6 Box Plot for Total Error Score per CP Attempt (Macro-Group) 


C. Map and Compass Checks 

Subjects had their map check scores calculated for each leg of the course. 
As mentioned in Chapter III Map Checks, subjects were allowed to make three different 
kinds of map checks during the course of the experiment. These were map only, compass 
only, or map and compass together. These checks were timed and recorded by the 
monitor. Subjects were not limited in the number of map checks they were allowed to 
take, and they could take back-to-back checks if they desired additional time. Subject's 
instructions were to "minimize" the number of map-checks, which clearly meant different 
things to different people. Some subjects refused to request the map, even if they realized 
they had left their route or when they were confused. Others requested the map at each 
control, even if they had been performing well. 

The weights of the mapcheck scores are the same as in the Goerger 
experiment to allow for comparison of results [GOER 98b]. Subjects who took a map or a 
compass check got one point. Map and compass checks combined were 1.5 points, since 
the two in combination allowed more information to be gathered. Subjects who desired to 
change their route were assessed a 0.5 point penalty. While additional information can be 


gained from the map during the one minute allotted to mark the new route, this is offset by 
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the relatively limited study time the individual has to plan a new route. Goerger stated that 
the subject must already have knowledge of their location and destination in order to 
correctly plan an effective new route [GOER 98b]. Accordingly, this value was set lower 
so as not to penalize the subjects for information they ideally already had. The marking of 
a new route from a misidentified location immediately gave the subject a parallel error 
when they started to move. If a subject left the course boundaries, they were given a two- 
point error because they demonstrated a lack of knowledge of that area. Knowing they 
moved out-of-bounds, they discover that their current mental map is in error. This lets 
them know that they are not where they think they are, thus giving information that may 
not be gleaned from looking at the map alone. Finally, if subjects had been lost for 15 
continuous minutes they were awarded a reorientation error with 3.0 points. This extreme 
penalty is because they were shown their exact location on the map, and were then given 
the opportunity to plan a new route to the next control. 

The lower the mapcheck score, the better the overall performance. Figure 
4.7 shows the box plot of the groups using a normalized map check. This number was the 
total number of mapcheck points the subject was awarded divided by the number of 


controls attempted. The means between training groups are statistically different with 
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Figure 4.7. Box Plot for Normalized Map Check Score per CP Attempt (Group) 
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F(6,35) = 3.078, P = 0.0193. There were significant differences between the means of the 
VEI, VE2, MAPI, and the MAP2 groups and the RW and VE groups. These results 
indicate two different things. First, in the case of the low-fidelity map groups, subjects 
tended to not consult the map as much because of the scarcity of detail on the map. Also, 
those subjects tended to choose sumpler routes that kept to the exterior boundary roads 
and thus had lesser chance to make parallel errors and need to make map consultations. 
This finding is explored in more detail in section B. Second, as noted in the original 
Goerger experiment, subjects who had more confidence in their mental maps required less 
checks to resolve differences. The map-only group is merely demonstrating the same 
effect as shown in the Goerger experiment [GOER 98b]. The results suggest that the 
train-to-standard VE group was also able to develop this superior mental map during 
training. This suggests that training-to-standard developed an individual’s mental map 
more than the hurried one hour study period previously allowed. 
6. Survey Knowledge 
As explained in Chapter II Spatial Knowledge, survey knowledge is considered to 
be the highest level of spatial knowledge and is the most difficult to acquire. In order to 
determine the subject's level of survey knowledge of the environment, three different tests 
were conducted. 
a. Wheel Test 
Chapter III, Section J. Wheel Test describes the conduct and reasoning 
behind the wheel test, which was conducted at controls two and four. All subjects in the 
experiments conducted for this thesis performed these tests. In the original Goerger 
experiment, four subjects failed to reach control point four and thus did not take the 
second wheel test. For the test, the subject’s answers were compared to the actual angle 
of the controls. The absolute value of the difference was calculated and summed for each 
test (CP2 and CP4 respectively). A lower angle difference equates to better performance. 
Figure 4.8 shows the total average angular difference based on the groups. Figure 4.9 


shows the group results for CP2 and Figure 4.10 shows the CP4 results. 
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Figure 4.8. Box Plot for Average Wheel Test Angular Difference (Group) 
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Figure 4.9. Box Plot for CP2 Wheel Test Angular Difference (Group) 
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Figure 4.10. Box Plot for CP4 Wheel Test Angular Difference (Group) 
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For the total average, the means between the groups show no statistical 
difference, F(2,35) = 0.425, P = 0.859. The same is true for the individual tests evaluated 
separately: CP2 (F(2,35) = 1.162, P = 0.354) and CP4 (F(2,31)= 2.488, P = 0.0514). 
Direct observation of the results suggests that overall there was no discernable pattern. At 
CP4, both the MAP2 and VE2 averaged the least angular difference of any of the groups. 
Perhaps the clear depiction of the depression resulted in a clearer mental map at this point. 

As a further check regarding survey knowledge, the data was examined at 
the macro-group level. Figure 4.11 shows the box plot of the results. All of the groups 
averaged a total of about 27 degrees off in their estimation. While not statistically valid 
(F(2,35) = 0.231, P =0.794 ), direct observation suggests that subjects who used only the 
map had less deviation in their answers, resulting in more uniform performance. This does 


not show that any group had better survey knowledge. 
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Figure 4.11. Box Plot for Average Wheel Test Angular Difference (Macro-Group) 


b. White Board Test 

Upon the conclusion of the experiment, every subject was given the 
whiteboard test (see Chapter III White Board Test). The analysis of the whiteboard results 
allowed a subject's level of exocentric survey knowledge to be evaluated. This was done 


by following the same procedures in the Goerger experiment (GOER 98b]. The angle 


between the control points was measured by determining the distance the subject judged 
the points to be apart in the XY plane. This distance was normalized using the total 
distance and then, using the Pythagorean Theorem, the angles between successive points 
were calculated. The subject angle results were compared to the actual course angles. 


Figure 4.12 shows a box plot of the resulting angular differences by group. 


Box Plot 
Split By: GROUP 


VEI 
VE 


[e] [e] [e] PB] [6] BU] IB 
= 





Avg Angle Diff 
Figure 4.12. Box Plot for Average Whiteboard Angular Difference (Group) 


Statistically there was no difference between the means with F(6,35) = 
0.789, P = 0.586. Direct observation seems to show that the real world had less variance 
between their answers, but by no means did they perform better than any other group. 


Figure 4.13 analyzes the results based on the macro-group. 
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Figure 4.13. Box Plot for Average Whiteboard Angular Difference (Macro-Group) 
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No statistical variation is noticeable (F(2,35) = 0.480, P = 0.624), nor do 
any differences appear in direct observation. The results seem to suggest that there is no 
difference between the groups in survey knowledge based on the angle error. 

The distance difference from the normalized distance of a leg and the user’s 
normalized distance from the whiteboard was also analyzed and compared. As with the 
angle analysis, no statistical differences were noted among either the groups (F(2,6) = 
2.19, P = 0.0732) or the macro-groups (F(2,35) = 1.37, P = 0.268). Figure 4.14 shows the 
box plot for the group average distance difference. Direct observation of the data shows 
that the MAP2 group averaged lower errors on this task. The VE] group also had the 
most variation, suggesting that the VE did not play a significant role in imparting survey 
knowledge. It also suggests that the lack of means of distance estimation in the VE 


particularly hurt those who spent extended time in the model. 
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Figure 4.14. Box Plot for Average Whiteboard Distance Difference (Group) 


Overall, there does not seem to be a large difference between the various 
groups in the survey knowledge evaluated by the whiteboard. This may be in part because 
this test was given at the very end of the course, after all individuals had been exposed to 
the area. This exposure may have equalized any benefit or hindrance imparted by the 
various training methods. It might be better for future experiments to conduct the 
whiteboard test prior to running the course, so that the subject will not have modified their 


mental map based on the actual terrain. 


С. Unplanned Route Execution 

If, and only if, the subjects successfully complete the course in one hour 
were they given the unplanned route test. For this task the subject would have to first 
explain how they would go from their current location, control point nine, to control point 
four. They had to give this explanation without first looking at a map. The subjects were 
then given ten minutes to move from CP9 to CP4. Most of the subjects planned this route 
to allow them to go over terrain they had been over once before, whether from a 
previously planned route, or from travelling through the area while in error. For the 
experiments done for this thesis, every subject who reached control point nine was able to 
successfully complete the unplanned route in the time allotted and none made errors off 
their planned route. This was also the result of the Goerger study [GOER 98b]. While 
there was a slight difference in the time subjects took to accomplish the task, it was not 
significant. 

These results seem to indicate there is no discernable difference in 
acquiring survey knowledge based on training method. However, because the test was not 
given to every subject, it is impossible to draw any conclusions concerning this task 
performance and training grouping. There is also the possibility that since subjects had 
actually traveled the ground, that any survey knowledge gained was the result of this 
direct exposure. 

One interesting caveat to this experiment: One test subject (M1-1) was 
timed out 51 meters short of CP9 and was on his route when time ended. Because he was 
so close to finishing, the subject was given the unplanned route test (though because he 
did not finish, his results were not included). Surprisingly, the subject was not able to 
accomplish the task. Instead, he become disoriented and missed his planned turn. Whether 
this result was due to an incomplete mental map or some other factor is unknown. 

T: Navigational Performance by Training Condition 

The results of the analysis do indicate statistical significance based upon the 
training condition. The results show that those who trained using the VE1, VE2, MAPI, 
and MAP2 generally performed landmark and route tasks better than the original VE and 


RW group. Direct observation of the results showed the train-to-standard groups 
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performed better than the Goerger training groups (RW, MAP, and VE) and the low 
fidelity map group (VE2, MAP2). Direct observation also showed that the train-to- 
standard VE group appeared to outperform all other groups in the route and landmark 
tasks. | 

Caution is advised when using these for conclusion for a number of reasons. The 
relatively small sample size in each group allows the results to be thrown off by one or 
two outlying results. Also, it is difficult to compare the results of broad categories of 
errors and distances given the vastly different routes subjects used. If subject A made his 
errors on a short but difficult route, how comparable is the mistake to subject B’s mistake 
on a longer easier route? Another reason for caution is that the standards for judging 
errors in the study required navigation to standards that are vastly different from most 
military subject’s experience. Most military navigators use terrain association and do not 
hesitate to perform quick detours to confirm they are on their route. In this experiment, 
this technique results in an error. Many subjects would also adopt a wide, circular search 
pattern for the controls, adding error distance. 

The low fidelity map group’s (MAP2, VE2) switching to the boundary road 
approach was unforeseen, and resulted in their better performance than the original 
Goerger groups. While travelling the boundary road was not always considered a simpler 
route (see Appendix L) it was easier to follow than the numerous twisting interior trails. 
Perhaps prohibiting use of the boundary roads would have produced results more in 
keeping with the second hypothesis of this thesis. However, because no such restriction 
existed for the other groups this would have made the results incomparable. 

With regards to survey knowledge there was no discernable difference between 
either the training groups or the macro groups. This may be in part due to the nature in 
which most of the tests were conducted. Since only those who finished the course were 
given the unplanned route test, only those with a better overall picture of the environment 
were able to attempt this test. It is no surprise that they completed it. The exposure to the 
actual course also may have blurred potential distinctions between the training groups in 
the whiteboard results. Future research may need to focus more on the survey task directly 


in order to better discern the differences, if any. 
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B. DISCUSSION 

1. Effect of Lowering Map Fidelity. 

As explained in Chapter III, Section N. Training, the second experiment was 
conducted to examine possible inherent bias in the highly detailed map given subjects of 
the earlier experiments. The expected results were that everyone would perform more 
poorly, but that the VE group’s performance drop would be less severe than that of the 
map-only group. As was noted earlier in Sections A.4, A.5, and A.6, the low fidelity map- 
only and VE groups did better than their corresponding groups in the original Goerger 
experiment. What would explain this? 

First, it should be noted that while the map had a lower overall fidelity, one thing 
the map improved upon was in clearly showing control point four was in a depression. 
Nevertheless, how much effect did this clarification have on performance? Figure 4.15 


shows the seven groups number of errors on control point four. 
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Figure 4.15. Box Plot for Number of Errors CP4 (Group) 


This figure shows that on average, the low fidelity map groups made one less error 
than the original VE group. In fact, the original VE group’s average for CP4 is twice that 
of their overall average (see Figure 4.3). This is a higher percentage than for the other 
groups. Both the MAPI train-to-standard and the MAP group also had much higher error 
rates for this control than their other points. Note that neither the low fidelity map, nor the 


extended exposure to the virtual environment for VEI, eliminated errors. The real effect is 
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seen in Figure 4.16, which shows the average amount of error distance for each group at 


СРА. 
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Figure 4.16. Box Plot for Average Distance per Error CP4 (Group) 


From these results, we clearly see that the original VE, MAP, and RW had much 
higher error distances at CP4 compared to the low fidelity map. This raised their overall 
total error score considerably. The train-to-standard VE group escaped this problem, most 
likely due to the extended exposure to the area during training allowing them to overcome 


the poor map representation. 
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Figure 4.17. Map Checks Score for CP4 (Group) 


Figure 4.17 shows the map check scores for CP4. From the box plot, the results 
show that every group other than the low fidelity map groups and the train-to-standard 
УЕ! group had much higher map checks score than average at CP4. Since the low fidelity 
group did not have this large statistical jump, their overall performance was lower. 

The other reason for the improved performance is that the low fidelity map groups 
adjusted their routes based on the information provided. Since this group was provided 
almost no information about the interior trails in the area, they tended to avoid internal 
travel whenever possible and stuck to the boundary roads more than the other groups, 
even for short.movements. While staying on the boundary roads was not always 
considered the easier route (see Appendix L) it did have the effect of reducing complexity 
and reducing the chance of parallel errors. There tended to be limited trail and path 
intersections with the boundary road, so individuals could easily discern their location. 
Since those intersections that did exist were prominent, the low fidelity group tended to 
have good attack points when they left the boundary road to begin their search. Figures 
4.18 and 4.19 show the box plots of the planned difficulty of the routes for the two 
groups. While there is no statistical difference, direct observation suggests that the low 
fidelity groups planned easier routes. Again, remember that using the boundary road was 
not always considered the easier route since in many cases it added significant distance. 


Section B.2. Route Complexity shows how simpler routes improve performance. 
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Figure 4.18. Box Plot for Lisp Planned Route Difficulty (Map Fidelity) 
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Figure 4.19. Box Plot for Banker Planned Route Difficulty (Map Fidelity) 


Many subjects were aware that their strategy of using the boundary roads was 
adding distance. Within VE2, both subjects who failed to find CP7 determined that they 
were “eating” too much time by using the boundary roads. They planned to incorporate a 
shortcut while enroute from CP6 to CP7 along the boundary road by cutting through the 
woods to save time. In both cases, the subjects never recovered from their “shortcut.” 

Figure 4.20 clearly shows that the low fidelity map group conducted less map 


checks as well. This was most likely due to the lack of detail on the map. 
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Figure 4.20. Box Plot for Normalized Map Check Score per CP Attempt (Map Fidelity) 
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Overall, the fidelity of the map had no effect on survey knowledge tasks. Figure 
4.21 shows the box plots of the average angular difference from the wheel test (discussed 
in Section A.6 previously). No difference is discernable. Likewise, there is no difference 


for CP2. This is shown in Figure 4.22. 
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Figure 4.21. Wheel Test Angular Difference (Map Fidelity) 
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Figure 4.22. Average Wheel Test Angular Difference CP2 (Map Fidelity) 


There is a Statistical difference between the means at CP4, however, F(1,31) = 
4.736, P = 0.0373. Observation of the data suggests that the 1:24,000 scale map users had 
less angular difference than those with the 1:5,000 scale map. Figure 4.23 shows this 


result. The low fidelity scale map users averaged ten degrees less difference than the other 


group. The likely explanation for this was the confusion that many 1:5,000 scale users 
suffered over the location of CP4. In mistakenly looking for CP4 on a hilltop, when they 
finally located CP4, they had very recently changed their mental map. This may have 


resulted in their overall worse performance. 
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Figure 4.23. Average Wheel Test Angular Difference CP4 (Map Fidelity) 


Examining the results of the whiteboard test, we see no statistical difference 
between those with the high fidelity map and the low fidelity map. Direct observation 


suggests that the low-fidelity map averaged slightly less error. 
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Figure 4.24. Average Whiteboard Angular Difference (Map Fidelity) 


2. Route Complexity 

As mentioned in the preceding section, the fact that the low fidelity map groups 
shifted to the boundary roads resulted in giving them better performance than the original 
Goerger groups. In many cases, following the boundary roads was not considered the 
"simplest" route. What correlation, if any, exists between route difficulty and 
performance? 

Figure 4.25 shows the regression plot of the planned route difficulty of both the 
Lisp and Banker scales versus the Landmark results (see Appendix L). Direct observation 


of the plots suggests that as the planned difficulty of the route went up, the subject's score 
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Figure 4.25. Regression Plot for Landmark Results vs. Planned Route Difficulty 


went down. These results are statistically significant with the lisp results F= 7.78, P = 
0.0086, and Banker F= 5.223, P = 0.0289. This suggests that subjects with “easier” routes 
were less likely to make errors they could not recover from, nor were they disoriented as 
much. It also suggests that “easier” routes were more readily memorized and recalled. 
Similar results occurred for the errors per control point attempted, with both scales 
achieving statistically valid results (Banker: F= 7.815, P = 0.0086, Lisp: F= 10.584, P = 
0.0026). In both cases, subjects with easier routes made fewer errors. This is shown in 


Figure 4.26, with the lower number of errors indicating better performance. 
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Figure 4.26. Regression Plot for Errors per CP Attempt vs. Route Difficulty 


The results of simple regression of the route difficulty versus both the normalized 
map check score and the total error score, while not resulting in statistically valid results, 
did suggest a similar pattern. Direct observation of the data seemed to show that those 
with easier routes made less map checks and traveled less distance per error than those 
following routes that were more difficult. 

These results indicate that the difficulty of the route that subjects planned did have 
an effect on their route and landmark task performance. They also provide anecdotal 
support for the first hypothesis. In all three experiments in this series, real-world and VE 
subjects with a one hour planning limit often ran out of time during their training before 
adequately planning their route. This was often caused by the subject actively exploring 
possible routes from the start, as opposed to solely using the map. Because they spent 
much of their time exploring, they often had extremely limited time to plan and memorize 
the final legs of their route. Often this led to them just drawing a “straight-line” between 
points, as opposed to finding an easier route. Such “straight line” routes were almost 
always among the most difficult, and, from the above results, the most error prone. Thus, 
the one-hour time limit for the VE/real-world tended to force harder routes, which in turn 
resulted in poorer performance. Removing the time constraint resulted in easier routes and 


better performance. 
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2.8 


The user’s route difficulty had no bearing on their survey task performance. 
Regression analysis of route difficulty versus angular difference resulted in no significant 
pattern. Nor were there valid differences in the whiteboard test. 

3: Effect of User Ability Level 

Individual differences can have a large effect on performance, and one of the major 
differences between the subjects in this experiment was their level of navigational 
experience. Subjects evaluated their skill level both written and by using a bar-line. The 
bar-line was used for our comparison analysis, as no subjects evaluated themselves as 
beginners (possibly due to bias against the word "beginner") with their words, but several 
fell into this category by the bar-line. 

Examining the landmark results based on the subjects, there is no statistically valid 
difference between the three ability levels, F(2,35)= 1.938, P = 0.1605. However, direct 
observation of the data, shown in Figure 4.27, suggests that those subjects in the expert 
category found more control points. The expert group also had less variation. Beginners 


averaged the least number of controls found, and the most variation. 
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Figure 4.27. Landmark Score (Self-Eval) 


Figure 4.28 displays both the errors per control attempted and the total error score 
for the user’s navigational ability level. The lower score represents better performance. 
Neither analysis resulted in statistically valid difference of means. Direct observation of the 


results suggests that expert classification subjects traveled less distance after making an 
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error before they realized their mistake. The data also suggests that beginners made the 


most errors per CP attempted. 


Box Plot Box Plot 
5 Split By: Self-Eval Split By: Self-Eval 
1.8 


В Beginner 


м Екрегі 


intermediate 


Units 





Total Error Score/Attempt Errors Per CP Attermpt 


Figure 4.28. Total Error Score/Attempt and Errors per CP/Attempt (Self-Eval) 


The results of the normalized map check scores, analyzed by self-evaluated ability 
group, are shown in Figure 4.29. There is no correlation apparent upon direct observation. 
Intermediate ability users had the widest variation. Beginners averaged the same map 
check score as experts, but for different reasons. Experts were less likely to be lost and 


require a map check. Beginners were less likely to realize they needed one. 
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Figure 4.29. Normalized Map Check Score/Attempt (Self-Eval) 
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No differences were determined in the survey knowledge of subjects based on their 
self-evaluated ability level. Nor was there a statistical difference in the difficulty level of 
the subjects chosen routes. 

4. Spatial Ability Post-Hoc Correlation 

Goerger found a significant difference in the performance between the high spatial 
ability subjects and the low spatial ability subjects [GOER 98b]. These results were 
confirmed by this experiment. Examining the landmark results based on the subject's GZ 
spatial ability showed a significant difference between the means based on performance, 


F(1,35) 2 8.221, P = 0.0072. These results are shown in the box plot in Figure 4.30. 
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Figure 4.30. Landmark Score (GZ Ability Group) 


Figure 4.31 shows the total error score per CP attempted for the GZ ability group. 
The groups show statistical significance, F(1,35) = 7.698, P = 0.0092, Subjects who 
scored higher on the GZ traveled less distance per error, meaning they realized their errors 
faster than low GZ subjects and recovered sooner. Direct observation of high GZ ability 
subjects suggests they made less overall errors, as shown in Figure 4.32. These results 
were not significant at F(1,35) = 3.393, P = 0.0557. Overall, the results suggest that high 
ability subjects made fewer errors, and recovered faster from their errors, than low GZ 
subjects. This in turn enabled them to attempt more controls and achieve higher landmark 


scores. The exact cause of their better performance, whether better memory, mental map, 
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route selection, or overall organization, is not precisely determined at this point. Future 


research should examine exactly why those with high spatial ability perform better. 
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Figure 4.31. Total Error Score/Attempt (GZ Ability Group) 
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Figure 4.32. Errors per CP Attempt (GZ Ability Group) 


High GZ ability subjects also had lower normalized map check scores than their 
counterparts by a statistically significant margin, F(1,35) = 4.728, P = 0.0465. This is 
shown in Figure 4.33. These results suggest that high GZ score subjects realized their 
errors quickly enough that they did not need a map to confirm this. There was no 
difference in the normalized non-error map check scores between the groups (F(1,35) = 


0.712, P = 0.4048). These results are shown in Figure 4.34. This result casts doubt on 


Goerger’s assertion that the low GZ map check score is a result of better memory or more 
confidence in their route. High GZ subjects averaged roughly the same number of 


maintenance checks per control as the low ability group. 
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Figure 4.33. Normalized Map Check Score/Attempt (GZ Ability Group) 
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Figure 4.34. Normalized Maintenance Checks/Attempt (GZ Ability Group) 


There was no difference in the map reading skills of the two groups, however 
analysis showed that the high GZ group did select less difficult routes. This is shown in 
Figure 4.35. This significant difference (Banker F= 7.93, P = 0.0081, Lisp F = 8.141, P= 
0.0074) suggests that high GZ ability subjects were able to recognize easier routes and 


simpler movements than their counterparts. They were able to discern which routes 


79 


offered easily recognizable features and landmarks that could be used as progress 
indicators. This allowed them to confirm their position as they went, helping to reduce 


their overall number of errors. 
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Figure 4.35. Banker and Lisp Route Difficulty (GZ Ability Group) 


With regards to survey knowledge, there was no difference between the GZ ability 
levels and the whiteboard results. There was no statistical difference between the wheel 
test angular deviations (F (1,35) = 1.364, P = 0.2513). Direct observation of the wheel 
test results suggests that high ability subjects did average slightly less error than their low 
ability counterparts. This is shown in Figure 4.36. This implies that high spatial ability 
allowed subjects to better orient their mental map of the environment on demand to 
determine the location of other landmarks in regards to their own position. 
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Figure 4.36. Wheel Test Angular Difference (GZ Ability Group) 
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5. Analysis Based on Santa Barbara 

Goerger, in his experiment, did not determine any discernable results based on the 
Santa Barbara sense of direction scale [GOER 98b]. The results of this experiment did 
show a significant difference between the results. Note, however, the very low number of 
subjects who were classified as low. | 

Figure 4.37 displays the landmark results based on the Santa Barbara scale. The 
results are statistically different with F(1,35) = 7.093, P= 0.0119. 
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Figure 4.37. Landmark Score (Santa Barbara H/L) 


Figure 4.38 shows the box blot for the errors per CP attempted based on the 
subject’s Santa Barbara results. The analysis of means shows a significant difference 
between the two classifications, F(1,35) = 19.240, P = 0.0001. Looking at the results of 
these two plots shows that those classified as high ability by the Santa Barbara averaged 
half the number of errors of the low ability group. Because they made fewer errors, they 
appeared less inclined to lose their way, and in turn, this helped them find more controls. 
Also shown in Figure 4.38 is the total error score. While these results are not statistically 
different (F = 2.333, P = 0.264), direct observation does suggest that the high Santa 


Barbara group traveled less distance per error before recovering. 
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Figure 4.38. Errors per CP Attempt and Total Error Score /Attempt (Santa Barbara) 


The data showed no statistically significant difference between the Santa Barbara 
levels and the normalized map check scores or the number of maintenance checks the 
subject made. However, there was a significant difference between the means in the level 
of route difficulty between the groups (Banker F = 4.513, P = 0.0412, Lisp F = 5.84, P = 
0.0243). This result is shown in Figure 4.39 with the lower score being the easier route. 
Observation of these results indicate that low Santa Barbara subjects tended to choose 
more difficult routes. This choice in turn may have caused them to make more errors. 
These subjects may have lacked intermittent landmarks to help them confirm they were on 


track, allowing them to travel further before they recovered from errors. 
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Figure 4.39. Banker and Lisp Route Difficulty (Santa Barbara) 


There were no differences between the groups for the wheel test survey tasks. 
Neither the high nor the low group performed better on the whiteboard. Only one low 
ability subject completed the course, but that subject was able to complete the unplanned 
route task to standard so no conclusions can be drawn there as well. There appears to be 
no relation between a subject’s survey knowledge and their Santa Barbara rating. 

6. Simulator Sickness 

Goerger had commented on user’s having simulator sickness during the conduct of 
the experiments [GOER 98b]. Of the ten subjects who used the VE in this experiment, 
only one complained of simulator sickness. That individual described feeling sick but said 
he attributed it to a lack of food prior to beginning the experiment. The subject had found 
he had to take frequent breaks until it became quite uncomfortable. The subject, who was 
part of the train-to-standard group, was given 20 minutes to go eat. Upon returning, all 
symptoms of simulator sickness disappeared and he was able to successfully continue the 
experiment without hindrance. No other VE subjects experienced any symptoms, though 
many spent considerable time training (one subject trained almost three hours). Remember 
that all of the train-to-standard subjects had to complete the straight run through of the 
course in the VE as part of the pre-course test prior to being allowed on the course. This 
would tend to suggest that simulator sickness is largely due to individual differences and 
the physical condition of the subjects at the time of the experiment. 

T Distinguishing Benefits Between Groups. 

The analysis of the data so far has shown that there are both differences between 
the spatial knowledge gained based on training group and based on spatial ability. The 
question that is often asked is whether one training method is better for training a 
particular type of spatial ability individual? In other words, is it better to train a low spatial 
ability person using a map or by the VE? In order to answer this question the results of the 
error per CP attempt and total error score were examined broken down by group and GZ 
ability. Table 1 shows the means of the errors per CP results. No significant statistical 
results can be concluded from the data. However direct observation of the data does 
suggest that there was a smaller difference between the results of the low ability and high 


ability subjects who used the VE than those who used the map or were real world 
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subjects. Table 1 shows that the low spatial ability map-only groups averaged 0.3 errors 
more per CP than the high ability GZ subjects in the same training group. The real world 
had an even larger difference, with the low ability averaging 0.6 errors more per CP than 
high ability subjects. In contrast, the low GZ VE subjects averaged only 0.1 to 0.2 errors 
more than their high GZ counterparts. This suggests that low spatial ability subjects 
perform closer to their high ability counterparts when trained using the VE at least as far 
as the number of errors committed. This may be because of the advantages high spatial 
ability subjects have in mentally rotating the map to envision what the terrain will look 
like, which low GZ subjects may lack. In the VE, subjects did not have to do this mental 


translation and thus all performed to roughly the same level. 


Descriptive Statistics 
Split By: Group, Ability Group 
Mean Std. Dev. Std. Error Count 
Errors Per CP Attempt, Total | .eeo|  331|  .056| 35 
Errors Per CP Attempt, MAP, High 
Errors Per CP Attempt, MAP, Low 
Errors Per CP Attempt, MAP1, High 
Errors Per CP Attempt, MAP1, Low 


Errors Per CP Attempt, MAP2, High 
Errors Per CP Attempt, MAP2, Low 
Errors Per CP Attempt, RW, High 
Errors Per CP Attempt, RW, Low 
Errors Per CP Attempt, VE, High 
Errors Per CP Attempt, VE, Low 
Errors Per CP Attempt, VE1, High 
Errors Per CP Attempt, VE1, Low 
Errors Per CP Attempt, V E2, High 
Errors Per CP Attempt, VE2, Low 





Table 1. Errors per CP Attempted by Group and GZ Ability 


Table 2 shows the descriptive statistics concerning the total error score per CP 
attempted. Again, no significant differences can be inferred. In this case, it is not readily 
apparent if any one style of training technique minimizes the differences between high and 
low GZ ability. The largest percentage difference is in the real world group, suggesting 
that while direct exploration for a limited period may be effective for high GZ subjects, 


low ability subjects would best avoid this technique. The subjects in the Goerger study had 
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the largest differences between the ability levels. The extended study time may explain the 
closer performance levels for the train-to-standard groups (VE1, MAP1), and the poor 
map and resulting easier route selection may do the same for the low fidelity map groups 
(MAP2, VE2). In these groups, there seemed little difference in how quickly the high and 
low spatial ability subjects realized they had made an error. Considering only the Goerger 
groups, while as a whole the map group realized their errors quicker, the difference 


between the spatial ability groups as a percentage was the smallest in the VE group. 


Descriptive Statistics 
Split By: Group, Ability Group 
Mean  Std.Dev. Std. Error Count 
|15381| 68507) 4842: 2 
| 25558!  15972|  11294| 2 
[26.200 | __ 8.084 | 4о2| 4 
1005 т 
| 17352! — 1074| 279) 2 
[30.081 | 8717] 644) 2 
Total Error Score/Attempt, VE2, High 
Total Error Score/Attempt, VE2, Low | 36.181| 29.443] 20.819] 2| 
Table 2. Total Error Score/Attempt by Group and GZ Ability 




















Total Error Score/Attempt, Total 

Total Error Score/Attempt, MAP, High 
Total Error Score/Attempt, MAP, Low 
Total Error Score/Attempt, MAP1, High 
Total Error Score/Attempt, MAP1, Low 
Total Error Score/Attempt, MAP2, High 
Total Error Score/Attempt, MAP2, Low 
Total Error Score/Attempt, RW, High 
Total Error Score/Attempt, RW, Low 
Total Error Score/Attempt, VE, High 
Total Error Score/Attempt, VE, Low 
Total Error Score/Attempt, VE1, High 
Total Error Score/Attempt, VE1, Low 





Overall, the results in this section suggest that it may be beneficial for low spatial 
ability subjects to train using the VE, as this may help to visualize the map in a way that 
they have difficulty doing on their own. Further research into this area is needed to 
determine if low ability subjects really would be better training by one particular method. 

8. Non-Error Map Checks 

During the course of the orienteering experiment, individuals would sometimes 
conduct map checks even if they were still on their planned path or at a known point. 
These non-error map checks are referred to as maintenance checks. Some of the 
individuals may have been disoriented at the time of the check (even if not yet in error). 


Most, however, used these checks to confirm their memory of their planned route or to 
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refresh themselves as to the features they planned to use to assist them in finding the next 
control. Some subjects had planned to request the compass at certain locations to conduct 
dead reckoning from a known attack point. As this style of map check is completely 
different from those caused by errors, the data was analyzed by considering only the 
maintenance map checks to see what effect, if any, the training group, user’s spatial 
ability, user's navigational ability, etc., had on their requesting maintenance checks. Figure 


4.40 shows the box plot results for the training groups. 
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Figure 4.40. Box Plot of Maintenance Checks (Group) 


The analysis of means is not statistically significant, F(6,35) = 2.038, P = 0.937 
Most groups averaged around three points worth of maintenance map checks. Direct 
observation of the results suggests that the train-to-standard VE group (VE1) averaged 
the least number of maintenance checks (0.7). This suggests that extended exploration of 
the environment in the VE allowed them to better internalize their route so that they 
needed fewer reminders during the course. The low fidelity map-only group also scored 
below the average. It is likely that the lack of information on the map persuaded subjects 
not to consult it. 

Examining the results based on ability group, there is no discernable differences 
between the number of maintenance map checks and either GZ ability level, Santa Barbara 


level, or the user's experience. Furthermore, regression analysis showed no association 
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between the number of maintenance checks and the subject’s resulting landmark score 


(F(1,35) = 0.550, P = 0.4637). Figure 4.41 shows this regression analysis. 
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Figure 4.41. Regression Plot of Maintenance Checks vs. Landmark Score 


Figure 4.42 shows the same analysis this time comparing maintenance checks and 
the errors per control attempted. Again, there is no correlation between the number of 


maintenance checks and the subject’s performance. 
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Figure 4.42. Regression Plot of Maintenance Checks vs. Errors per CP Attempted 


From the above, we can conclude that while the training group may have an effect 


on the number of maintenance checks a subject takes, the number of maintenance checks a 
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subject takes does not appear to affect their actual performance. The number of 
maintenance checks appears totally independent from the subject’s ability level. 

9, Correlation Between Disorientation in Virtual Environment and 

Disorientation in Real World 

In his 1998 study, Goerger noticed a possible correlation between subject’s 
disorientation in the VE and in the same location in the real world [GOER 98b]. The same 
general effect was noticed on several occasions in this experiment as well. Subject VE2-1 
had trouble in the VE looking for CP3. The subject in fact bypassed it on the left and 
continued into the depression near CP4 before realizing the error. Despite resolving this in 
the VE, the subject made the exact same error while running the course, even remarking 
“Shoot, I did the same thing in the computer”. No concrete conclusion can be drawn, 
however, because of the limited number of participants who demonstrated this effect. 

Generally, subjects were asked which point they found to be the hardest to find in 
the VE and which they felt would be the toughest in the real world. Some of the subject’s 
difficulties came from the limitations in the VE model (cannot see through/under brush). 
Many subjects assumed CP5 would be difficult to find based on their model experience. 
Several subjects also were very up front about their planned usage of trail intersections for 
navigation. Many commented that if they could not discern those intersections on the 
actual course, they would be in trouble. 

Several subjects got into trouble by planning cross-country movements using the 
VE. During the course of the VE training, they relied heavily on the compass to cut direct 
movement while “beating brush.” The ease of this method in the VE disarmed the subjects 
to the actual difficulty of their planned route, and the subjects had difficulty replicating 
these routes on the actual course. 

10. Resolving Ambiguities in Mental Maps 

One of the biggest difficulties facing the subjects was resolving differences in their 
mental maps upon arrival on the ground. These difficulties were noted by Goerger in his 
experiment, and similar effects were observed during this experiment. Goerger felt that 


each group (map-only, VE and real-world) faced it’s own specialized challenges in this 
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regard. These differences are partly due to the way the information bout the environment 
was acquired. 

VE (and real-world) subjects gathered much of their information from dynamic 
imagery. Many VE subjects described it as playing back a movie in their heads. They 
would try to match the imagery their eyes were seeing to this mental movie in their heads 
to determine if they were on course. The potential pitfall in this, as discussed by Goerger 
[GOER 98b], is when this mental movie is incomplete or inaccurate. Then users, as 
opposed to confirming their location, merely become confused and disoriented. 

Map-only subjects tended to store their information as a list of objects or actions 
referred to as static imagery or symbolic imagery. This gave them knowledge about 
particular points of interest but fuzzier information about indistinct areas. This in turn 
lends the map-only groups more slack when conducting the course, as they are only 
looking for a few key points of interest. Map-only subjects became confused only when 
they encountered key structures out of turn or encountered areas that appeared similar to 
their focus area (see Chapter III Errors). Many map-only subjects discarded distance 
estimation entirely, relying solely upon their mental checklist. While this may not have 
assisted them, it did not hinder their completion of the course unless they later tried to 
apply distance estimation to their mental model. Subjects who stuck to a poor strategy 
performed better than those who tried to change tactics "in the middle." This later group 
usually only further confused themselves and ended up becoming totally disoriented. 

For all of the groups, it was critical that they quickly adjust their mental models to 
the terrain. This meant transferring the picture/movie in their minds into actual distances 
covered on the ground. Once subjects established the distance/scale relationships, they 
usually had little difficulties later. The caveat is that subjects with imprecise mental models 
often radically change their entire model, not just the bad portions. One map subject who 
did not memorize distances, underestimated the distance between the SP and CPI, as well 
as CP1 and CP2. Having discovered that these points were much closer than he expected, 
he radically shorted his mental map image of the area. This resulted in his stopping two 
hundred meters short of CP4 when he first initiated his search. Because the first two 


controls had been closer than he thought, he guessed that the whole course was that way. 
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The VE group in particular had the most trouble in quickly resolving differences 
between the real world and their mental model. The first movement in the course, from the 
SP to CP1, was where VEI subjects averaged their greatest amount of error distance. 
Figure 4.43 shows the distance per error for leg one for all groups. The lack of an 


effective means of discerning scale and distance traveled in the VE likely contributed to 


this early difficulty. 
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Figure 4.43. Distance per Error SP to CP1 (Group) 


While no statistical significance can be drawn, direct observations show that the 
real-world group made the least error distance on this movement. Also, disregarding one 
Goerger VE subject who circled the course area looking for CP1, all of the VE groups on 
this leg performed about the same regardless of training method. Once the mental 
ambiguities were resolved, however, the train-to-standard VE group was able to perform 
better on the rest of the course. 

11. Map Reading 

Goerger did not determine any difference in performance based on map reading 
ability. The results in this experiment did find some noticeable effects on performance 
based on map reading ability. For a subject’s landmark scores, the map reading score 
showed a significant effect, with F(1,34) = 7.143, P = 0.0116. Figure 4.44 show the 


regression plot. 
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Figure 4.44. Regression Plot of Landmark Score vs. Map Reading Score 


Observation of the data suggests that while a high map reading score is not a good 
predictor of success, a low map reading score is a good predictor of failure. Those 
subjects who had difficulty with basic map reading skills in turn had difficulty in locating 
the control points. 

Direct observation of the data suggested that subjects with poorer map reading 
skills also tended to make more errors per CP attempted and traveled more distance per 
error. This may be because they cannot identify viable routes, or the necessary features to 
help keep them on their route. Neither regression was significant statistically however (for 
the errors per CP attempted F= 3.916, P = 0.0562, distance per error F = 2.701, P = 


0.1098). The regressions for both are shown in Figure 4.45 below. 
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Figure 4.45. Regression Plots of Map Reading vs. Total Error Score/Attempt and 
Map Reading vs. Errors per CP Attempt 
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A subject’s map reading skills also may have had an effect on the difficulty of the 
route they chose. While no statistically valid difference exists, direct observation (see 
Figure 4.46) of the regression comparison suggests that subjects with higher map reading 


skills choose less difficult routes. 
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The map reading scores were examined to see how much they were influenced by 
the subject’s individual differences. There were no significant differences between the map 


reading scores of high and low GZ ability subjects. This is shown in Figure 4.47. 
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Figure 4.47. Map Reading Score (GZ Ability Group) 


There was also no difference between the Santa Barbara ability groups. There was 
an observed difference based on the bar-line evaluation. While those classified as experts 
did not outperform the others, those classified as beginners did demonstrate weaker map 


reading skills. Since map reading is largely a learned skill, this is to be expected. The data 


is Shown in Figure 4.48. 
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Figure 4.48. Map Reading Score (Self-Eval) 


Examining the distribution of map-reading scores across the groups, we find that 
the lowest scoring map-readers were in the original VE group. See Figure 4.49. This 
brings up an interesting question of whether or not the subject scored low because he 
could not read a map, or because of the training tool he used? The data may have to be 
analyzed using MANOVAs to answer which criteria had the strongest effect on 


performance. Exp 
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Figure 4.49. Map Reading Score (Group). 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 
1; General Conclusions 

The thesis experiments studied the effects of various training methods on 
the transfer of spatial knowledge to a real world environment. The first experiment used a 
high-resolution 1:5,000 scale map and the high fidelity real time VE. The subjects either 
trained only using the map or with the map and VE together. Subjects had to meet a set 
standard before continuing on to the actual testing phase. The second experiment used a 
lower fidelity 1:24,000 scale map but the same high fidelity VE. Subjects were again either 
map-only or VE and map together. Training time was limited to exactly one hour. The 
results of the experiments were then analyzed against each other and the original Goerger 
results [GOER 98b]. The following conclusions are drawn from both the quantitative and 
qualitative results previously presented: 

a. Subjects who trained to standard with a high fidelity VE 
demonstrated superior route and landmark knowledge to any other training group 
(Chapter IV, Sections A.4 and A.5). 

b. Spatial Ability (including Santa Barbara level or self-assessment 
of spatial ability) plays a significant role in an individual’s ability to obtain and 
demonstrate spatial knowledge (Chapter IV, Sections B.4 and B.5). High spatial 
ability subjects perform better on route and landmark tasks than do low spatial 
ability subjects. 

c Subjects with poor map reading skills are likely to have 
difficulty in navigational tasks and demonstrate significantly poorer spatial 
knowledge than any other subjects (Chapter IV, Section B.11). 

d. The difficulty of the planned route has an effect on its successful 
completion, and a significant effect of the amount of spatial knowledge gained 
(Chapter IV, Section B.2). The easier the route, the better the overall 
perfromance. 

е. Adjusting the fidelity of the map causes subjects to adjust their 


planned routes to the information that is provided (Chapter IV, Section B.1). The 
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accuracy of the map in representing terrain can have a dramatic effect on errors at 
specific locations (Chapter IV, Section B.1). 
2. Performance by Study Group 
Based on the outcomes of the three experiments, the results suggest that 
those who trained to a standard and not to time, gained more spatial knowledge about a 
one km square piece of terrain than those who trained to a one-hour time limit. VE 
subjects who trained to standard had the best overall landmark and route knowledge 
performance. Subjects who were given only one hour to train and a high fidelity map 
performed better than those with only an hour, the high fidelity map, and the high fidelity 
VE [GOER 98b]. Results suggest that with only one hour of training and a less detailed 
map (but with one critical area being more accurate), the subjects would adjust their 
routes to the detail provided. They then performed better in following these simple routes 
than those who planned their routes on the more detailed map. 
3. Performance by Spatial Ability 
The results suggest that regardless of training group, individuals with 
above average spatial ability scores (as determined by the Guilford-Zimmerman test) 
demonstrated more spatial knowledge than individuals with low spatial ability scores. 
Subjects with high Santa Barbara ability scores also outperformed those with low Santa 
Barbara ability scores. Higher ability level individuals selected easier routes and executed 
them better. The results suggest that high ability individuals had better egocentric 
knowledge than their counterparts. High spatial ability individuals demonstrated slightly 
better exocentric knowledge (survey knowledge) than low ability individuals. Using the 
train-to-standard VE seemed to minimize the differences between spatial ability users 
(Chapter IV, Section B.7). 
4. Performance by User Experience/Map Reading Skills 
Users who classified themselves as expert navigators made fewer errors 
than intermediate or beginner level subjects did. They also conducted fewer map checks. 
While they did find slightly more controls, this was not significant. There was no 
significant difference between ability levels in how fast subjects recognized their errors. 


Nor did subjects demonstrate differences in survey knowledge based on their experience 
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classification. Beginners were much less likely to make map checks, because they did not 
realize they needed them or were uncertain as to their benefit. Beginners also had much 
lower map reading skills on average. 

While a high map reading score was not a good predictor of success, a low map 
reading score was an excellent predictor of failure. Subjects with poor map reading skills 
had difficulty planning executable routes, had problems locating the controls, were more 
apt to make errors, and were less likely to recover quickly from those errors. 

S: Route Selection 

Subjects who planned simpler routes made fewer errors and found more 
controls than those who planned harder routes. These simpler routes included more key 
features that could be used as reference points or navigational aids. This helped them 
recover from errors they did make faster than other subjects. They avoided areas where 
parallel errors were likely. 

6. Map Fidelity 

Subjects who were given the low fidelity map adjusted by planning much 
simpler routes in general and by sticking to the boundary roads whenever possible. The 
importance of accuracy in the map was highlighted in the improved results in finding CP4. 
The original Banker map, however detailed, was not clear in showing this depression, 
which caused more errors for those who relied on that map. Subjects who had access to 
highly detailed maps and had time constraints gained more spatial knowledge than their 
VE counterparts [GOER 98b]. 


B. SIGNIFICANCE 
1. Study Group 
If mission constraints include limited preparation time and highly detailed 
maps, individuals who only use maps will gain more information about the mission area 
than those who also use the VE. If time is not a constraint, individuals who use the VE 
will gain more information than those who only study the map. Individuals should tailor 
their learning method based on the time available. If there is not enough time to 


thoroughly explore the VE and make multiple iterations, then stay with the map. This 


97 


basic rule will help individuals focus their efforts where they will make the most gain. If 
time is limited and only poor maps are available, using the VE may also prove valuable as 
long as there is information in the VE that is not included on the map. 

D: Spatial Ability 

High spatial ability individuals are better suited as navigators than low 
spatial ability individuals. These individuals should be identified if possible and be assigned 
in positions requiring navigational skill. If it is possible to improve an individual's spatial 
ability, it should be done. Low spatial ability individuals may have an easier time using the 
VE in training than just the map alone. This may be due to their difficulty in mentally 
orientating and visualizing a map. 

3i User Experience 

Experienced navigators make less errors and are more confident, if not 
actually better, at navigational tasks. The only way to become experienced is to practice. 
Individuals must conduct regular navigational practice across a variety of terrain types and 
using all navigational techniques to stay proficient. Navigation is best learned by doing. 
Individuals may also get involved in recreational activities that utilize navigational skills 
(Sport Orienteering, Scouting) to practice and have fun at the same time. 

Map reading is a critical and perishable skill. Individuals who cannot read a map 
will have difficulty performing even the most basic of navigational tasks. It is for this 
reason that basic map reading is the first navigational skill taught to soldiers. Individuals 
should plan to refresh their skill in this area prior to missions. 

4. Route Selection 

Good route selection is critical to navigational performance. Individuals 
who plan routes with numerous easy-to-remember reference points or simplified paths are 
much more likely to execute them. Mission constraints may prohibit the use of the most 
direct or the easiest route. Individuals should try to tailor their route difficulty to the 
mission. Difficulty for difficulty’s sake is not desirable. A good route should include many 


of the navigational aids described in Appendix P. 
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Э; Map Fidelity 
Maps, regardless of how detailed, must accurately show the terrain in a 
clear manner. A map with too much detail clutter may actually induce error at certain 
locations. An accurate high fidelity map can be relied upon for success if time is limited. A 
low fidelity map will force subjects to simplify their planned movements to use the 
information provided by the map. A high fidelity map may offer users more freedom as 
they can have multiple reference points to choose from when planning their route. 
Ideally, a high fidelity VE has the same information as a high fidelity map. The VE 
takes longer to build and is not as easy to reproduce nor can it be carried with you. Thus 
regardless of the time allowed for preparation, a map should always be included in the 


program. 


С. FUTURE WORK 

While this thesis may have validated the utility of using a VE to transfer spatial 
knowledge about a natural area, there remain significant areas for future work and 
exploration. 

1. Immersive vs. NonImmersive 

Ruddle, Randle, Payne, and Jones [RUDD 96] concluded that there was no 

significant difference between immersive and non-immersive VEs with regards to 
performance results. However, their experiment was conducted in an indoor environment 
that was relatively small in scale. Will the same conclusion hold to a large outdoor 
environment? Most VE subjects, judging by their survey answers, felt comfortable with 
the three screen wide FOV display. However, this may not be the most immersive, nor the 
best way to display the VE. Future experiments can compare the current VE results to 
those obtained using a more immersive interface. The MOVES curriculum at NPS is 
currently building their own VE CAVE that could be used in the experiment in addition to 
head mounted displays. The effect of downsizing the wide screen TVs to a more 


affordable and space friendly three monitor configuration should also be studied. 
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2 Incorporating Sense of Scale into the VE 

One comment heard repeatedly from the VE subjects was how they lacked a real 
sense of scale in the VE model. They complained of being unable to accurately estimate 
distance and often felt uncomfortable when first introduced into the real world until they 
were able to focus their mental map. This result backs up the findings of Witmer and Kline 
[WITM 98] in that judging distance in a VE is difficult, and usually underestimated. 
Goerger had originally added two HMMWYVs to the model at the start point in order to 
give a sense of scale [GOER 98b], but most subjects didn't find this particularly useful. 
There are several possible options for introducing a sense of scale and distance traveled 
into the model. It would be possible to incorporate some sort of sound cue that could beep 
after a set distance of travel. A visual pace count or pedometer could also be displayed. 
Users should be able to reset this counter easily as they maneuver through the 
environment. Another option would be to change the interface into one that actually 
requires locomotion by the subject, whether by treadmill or cycle. While these last two 
methods may negate the advantage of being able to explore faster than humanly possible, 
it would allow users a more realistic sense of traveling and to acquire a sense of distance 
traveled. 

3. Fidelity Levels 

The current model is classified as high fidelity (see Chapter III, Section C Model). 
Now that the transfer of spatial knowledge from VE to natural terrain is a proven concept, 
new experiments could adjust the fidelity level of the model to determine what is both the 
optimal level of fidelity for transfer, and what is the minimum level of fidelity to achieve 
this transfer. Hopefully, reducing the fidelity would enable the model to be run on a less 
powerful machine. By determining the optimal levels, we can ensure we develop models 
that reach this but do not go past the point of diminishing returns. The model currently 
uses digital photographs to create photo-realistic bushes, trees, and buildings. However, 
all this comes at the expense of processing power. What if the same training effect can be 
achieved by substituting green cones for trees? Is it really an advantage to exactly replicate 
the buildings with detailed textured photographs or would a simple black square be 


enough? Obviously the idea of just incorporating symbolic representation (as is done with 


a map) may make the model appear less "realistic", but would that necessarily lessen the 
training impact? 


4. Transfer to a Desk-Top Model 


The current model requires a high-end multi-processor machine to run. This 
machine is both expensive and takes up considerable space. It would be of tremendous 
interest to the Army to determine if the model could be ported to a smaller desktop 
machine. Obviously, this would result in changes to both the display and to the model 
fidelity. However, these reductions would be accompanied by a cost and space savings. If 
such a transfer were proven feasible, it would be a boon to the military and would allow 
for a more widespread distribution of a VR training system. 

5; Design a MOUT Environment 

It has been shown that participants can gain spatial knowledge of indoor man- 
made environments, and in natural outdoor areas. The next logical step is to explore the 
knowledge transfer in a combination of these environments, namely urban areas. The 
perfect setting for this is available locally with the old MOUT site (Military Operations in 
Urban Terrain) on Fort Ord. This site consists of a several mock city blocks with multiple 
story cinderblock buildings with rooms, windows, staircases, and sewers. The area has 
streets and parks as well. The military used the site to practice fighting in cities and other 
built up areas. The area is currently used almost daily by local and national law 
enforcement agencies to train quick reaction forces or SWAT teams [BELC 99]. The 
military uses a standard design for its MOUT training areas, so detailed blueprints should 
be readily available to facilitate accurate model construction. Given the high usage of the 
area by local law enforcement, a large pool of test subjects is potentially available. It 
should be possible to arrange for test subjects and develop the standards for evaluating the 
models' effectiveness to everyone's satisfaction. 

6. Make the Model More 3D/ More Realistic 

At a recent virtual reality conference held at NPS, many psychologists complained 
about the use of billboard photos for trees or other non-flat objects. They said this 
technique robs the environment of its three-dimensional feel and makes it seem less real. 


Coupled with the fidelity research above, future work could focus on making the trees and 
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other objects seem less flat and more solid. Another problem that presented itself in the 
model was the lack of the ability to raise or lower the avatar's viewpoint from the set 5'8" 
height. While the 15-meter pop-up view was provided, this still did not allow the user the 
flexibility that they possess in the real world. Often when searching for a point in the VE, 
subjects desired to look under branches but could not lower themselves down without just 
backing up. But, in dense areas, backing up would usually put a new tree in front of them 
further obscuring the view. The ability to lower the avatar's viewpoint might allow for 
more "realistic" training. A third area for work is in collision detection. The current model 
has no collision detection and allows users to walk unimpeded through the trees and 
buildings. One potential problem with this, seen in a couple of VE subjects, was the 
selection of routes based on this "easy" movement. Upon arrival into the real world, these 
subjects regretted choosing a cross-country route through dense brush that was easy in the 
model but difficult and time-consuming in the real world (see Figure 5.1 Tight Squeeze). 
Adding some code to the model to slow movement through difficult terrain (whether 
dense woods, swamps, etc.) would make exploration in the model more realistic. It would 
also make it possible to allow weather effects to be incorporated into the model later (such 


as Slowed movement from mud or snow). 





Figure 5.1. Tight Squeeze 
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7. Cooperative Learning 

Within the educational community, many groups are reporting that computer 
trainers work best when cooperative learning is incorporated into the learning program. 
The optimal size seems to be two to three students [ROSE 95]. Rose concludes that 
* .. virtual reality technology that fosters collaboration will yield even greater educational 
benefits"[Rose 95]. The effect of cooperative learning could be done so two subjects 
could conduct the training phase together but then be tested in the environment separately 
or together. The results would be to see if they performed better as individuals than those 
who trained alone. This would also be more realistic because in the active military most 
navigation is done by a group, not an individual. Individuals would pool knowledge to 
arrive at a more informed decision. Obviously, introducing cooperative learning would 
also introduce a host of other factors, not the least of which include individual personality, 
compatibility, and cooperation. 

8. Training Navigation as a Secondary Task 

Land navigation is never the end goal of any military navigation other than in 
training courses. However, it is an essential skill to the successful completion of most 
missions. Given its critical, but secondary, nature, it would be interesting to see if spatial 
knowledge could be obtained while ostensibly not training navigation but another task. An 
ideal experiment could use a popular 3D-shooter type game such as Quake or Doom. 
Subjects would be told they are conducting research in virtual environments, not in land 
navigation in particular. They would be presented with the game environment built to 
replicate the outdoor Fort Ord area. The players would be given limited ammunition and 
have the goal of killing all the enemies. At each control point, additional ammo is found, 
as well as one or more enemies. However, to encourage the players to visit the controls in 
order, the program spawns additional enemies at the controls based on a time factor, with 
more enemies being produced at the lower number controls quicker than at the higher 
number. Once a control is captured however, it stops producing enemies. In this manner 
players would optimally clear the lower controls. Otherwise they may be swamped with 


enemies and be unable to "win". After several sessions, the players would be taken to the 
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outdoor environment and tested on the navigational tasks to see what knowledge was 
transferred during the course of their gaming. 

9. Evaluating Navigation as a Secondary Task 

As with Section 8 above, land navigation rarely stands alone. To provide validity 
of the model’s effectiveness, the evaluation phase should incorporate some secondary task 
that requires constant thinking, but is not totally overwhelming. A possible solution is the 
introduction of a hunter. As with most military mission, there is an enemy to be avoided. 
This person is armed with a paintball gun and is actively searching for the subject. Our 
subject is told that person has a map of the area but does not have the location of the 
control points. Our subject is not forced to follow a route, but is given an empty gun with 
ammunition being at the controls. He is also required to punch the stamp at each control 
point (to confirm he visits there). If he shoots the hunter the hunter must return to the start 
point (or so he is told). The goal is to reach every control point in the fastest time, while 
minimizing the number of times you are shot. 

10. Focus More on Survey Knowledge 

The research into VEs has shown it to be most effective in attaining route and 
landmark knowledge (see Chapter II). This is not to say that survey knowledge cannot be 
derived from a VE, just that it is not transferred as readily as the other forms. In this 
thesis' experiments, survey knowledge was not tested until after exposure to the 
environment, which may have produced additional learning and corrupted the results. To 
more accurately test the ability of the VE to transfer survey knowledge we must test 
survey knowledge more extensively during the experiments. One avenue to explore is to 
force the subject to change their route during the course of the experiment. By informing 
the subject of a chemical spill/forest fire and making them adjust their route without 
looking at a map, their survey knowledge could be tested earlier than the unplanned route 
test in the current model. Goerger suggested that another way of doing the same task 
would be to reduce the number of points but upon arrival at the test site force the subjects 
to run their route backwards [GOER 98b]. Either method would be more preferable to the 
current tests which were not done by all subjects, and certainly weren't done prior to 


extensive exposure to the area itself for an extend period of time (at least 30 minutes). 
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11. Introduce Competition into Training 

As mentioned above in Section seven, people often learn best when they do not 
learn alone. In this case, however, they will not be learning as much as in competition 
during their training. Using a networked virtual environment, two subjects would be run 
through simultaneously. Both subjects would start at the start point and race each other 
for the controls. Each would get a point for each control they "captured" first. Each 
control could be captured only once and must be captured in order. This would put 
pressure on the subjects to quickly decide upon a route and not get lost en-route. The 
game could be run several times. The subjects may or may not be given a familiarization 
phase with the area first. This experiment would not only examine the effects of 
competition, but would also make the navigation a secondary goal of the training. 

12. More Guided Instruction 

Another lesson in the learning community is that “Concrete teaching is necessary 
before students attain basic knowledge and skills to make free exploration" [FENG 95]. 
Feng notes that when students are given a "sea of resources" and told to sink or swim, 
many will sink. Without experience to guide them, many students will not learn efficiently 
as they do not know how to learn. With VEs, most, if not all, of the subjects had little 
experience with 3D models and environments. With a map, most subjects allowed for 
minor variances between what the map showed and they saw in the real world. Some 
subjects did not give the VE the same leeway. More than one VE subject fully expected all 
the trails in the real world to look like the *yellow brick roads" of the VE. Other subjects 
ignored potential reference points shown in the VE because they were not on the map and 
were uncertain how much they could “trust” the VE. Until people become more familiar 
with VEs, it may be necessary to guide subjects through the instruction of this unfamiliar 
medium more than the simple 15-minute orientation that was conducted in the 


experiments so far. 
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APPENDIX A. EXPERIMENT OUTLINE 


1) In Brief/Consent Form 


2) 


3) 


4) 


a) 
b) 
с) 
d) 


Time — 5 Min 

Location — CS Student Conf. Room 

OIC — CPT Quay B. Jones 

Materials — Consent Form, Privacy Act Statement, Minimal Risk Consent Form, 
Subject Roster, pencils, Fort Ord Map (confirm the subject has not been on the 
course terrain before), In Briefing Script 


Color Blindness Test/Self Evaluation Questionnaires/Map Reading Test 


a) 
b) 
с) 
d) 


е) 


Time — 15 Min 

Location — CS Student Conf Room 

OIC — CPT Quay B. Jones 

Materials — Color Charts (1 min), Self Ability Evaluation Sheet (1 min), Santa 
Barbara Sense of Direction Scale Questionnaire (3 min), Map Reading Test (5 
min), pencil 

Grading (5 min) 


Spatial Orientation 


a) 
b) 
с) 
9) 
е) 
f) 


Time — 15 Min 

Location — CS Student Conf Room 

OIC — CPT Quay B. Jones 

Materials — Guilford-Zimmerman Aptitude Tests (10 min), pencils, answer sheets, 
Grading and Grouping (5 min) 

Groups 


1) Group A - Upper 50 percentile 
ii) Group B - Lower 50 percentile 


Interface Familiarization (VE Only) 


a) 
b) 
с) 
d) 


е) 


Time — 15 Min minimum 

Location — Graphics Lab 

OIC — CPT Quay B. Jones 

Materials -SGI machine, Performer Town Model, Flybox instructions, Virtual 
Environment Briefing Script, Interface Familiarization Checklist 

Movement (15 min minimum) 
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5) Training 


a) Map Group (Low Fidelity Map) 

(1) Time - 60 Min 

(2) Location — CS Student Conf Room 

(3) OIC — CPT Quay B. Jones 

(4) Materials — 1:24,000 Scale Fort Ord Map, Aerial Photograph, Participant 
Task List, Map Marking Instructions, red alcohol marker, alcohol marker 
eraser, pencil, scratch paper, orienteering clue sheet, Low Fidelity Map 
Group Briefing Script, Training Evaluation Sheet 


b) Map Group (Train-to-Standard) 

(1) Time - Varies 

(2) Location — CS Student Conf Room 

(3) OIC — CPT Quay В. Jones 

(4) Materials — Fort Ord Orienteering Map, Participant Task List, Map 
Marking Instructions, red alcohol marker, alcohol marker eraser, pencil, 
scratch paper, orienteering clue sheet, Train-to-Standard Map Group 
Briefing Script, Training Evaluation Sheet 


c) Virtual Environment Group (Low Fidelity Map) 

(1) Time - 60 Min 

(2) Location — Graphics Lab CPT Quay B. Jones 

(3) OIC — CPT Quay B. Jones 

(4) Materials — Elvis (SGI ) w/ flybox and 21”/40” screen configuration or 
projector, Fort Ord Model, 1:24,000 Scale Fort Ord Map, Aerial 
Photograph, Participant Task List, Map Marking Instructions, Flybox 
instructions, red alcohol marker, alcohol marker eraser, pencil, scratch 
paper, orienteering clue sheet, Low Fidelity Map Virtual Environment 
Briefing Script, Training Evaluation Sheet 


d) Virtual Environment Group (Train-to-Standard) 

(1) Time - Varies 

(2) Location — Graphics Lab CPT Quay B. Jones 

(3) OIC — CPT Quay B. Jones 

(5) Materials — Elvis (SGI ) w/ flybox and 21”/40” screen configuration or 
projector, Fort Ord Model, Fort Ord Orienteering Map, Participant Task 
List, Map Marking Instructions, Flybox instructions, red alcohol marker, 
alcohol marker eraser, pencil, scratch paper, orienteering clue sheet, Train- 
to-Standard, Virtual Environment Briefing Script, Training Evaluation 
Sheet 
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6) 


Testing (est Time 120 Minutes — travel to Fort Ord Orienteering Course, run the 


course, and return). 


a) 
b) 
с) 
d) 


e) 


Time — Travel Time 30 Min (total); Run Course 90 Min; Total Time (120 min) 
Location — Fort Ord Orienteering Course 

OIC — CPT Quay B. Jones 

Materials — Clipboard with subject's map & designated route, compass, Think Out 
Loud Instructions, Data Collection Sheet, red pen to record data, blue alcohol pen, 
stop watch/timer, Color Wheel for Tasks 3.1. & 5.1, White Board with ten 
magnets, rucksack frame w/GPS system, helmet & camera, water, first aid kit 
(cellular phone), Course Briefing Script, blind fold (for movement to course), 
spare clue sheet & color wheel arrows, Tecnu (for poison oak) 

Tasks: 





i) Task 1. (Path Knowledge) Move from the starting point to Checkpoint #1 
along designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 

il) Task 2. (Path Knowledge) Move from Checkpoint #1 to Checkpoint #2 along 
designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 

ii) Task 3.1. (Survey Knowledge) Take bearings to SP, CP #5, and CP #9 at the 
south side of CP #4) 

iv) Task 3.2. (Path Knowledge) Move from Checkpoint #2 to Checkpoint #3 
along designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 

v) Task 4. (Path Knowledge) Move from Checkpoint #3 to Checkpoint #4 along 
designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 

vi) Task 5.1. (Survey Knowledge) Take bearings to CP #1, CP #6, and CP #8 at 
the south side of CP #4) 

vii) Task 5.2. (Path Knowledge) Move from Checkpoint #4 to Checkpoint #5 
along designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 

vill) Task6. (Path Knowledge) Move from Checkpoint #5 to Checkpoint #6 along 
designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 

ix) Task 7. (Path Knowledge) Move from Checkpoint #6 to Checkpoint #7 along 
designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 

x) Task 8. (Path Knowledge) Move from Checkpoint #7 to Checkpoint #8 along 
designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 

xi) Task 9. (Path Knowledge) Move from Checkpoint #8 to Checkpoint #9 along 
designated route. (measure elapsed time and number of errors; mark 
deviation from route on map) 
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7) 


xii) Task 10. (Survey Knowledge) Have subject indicate bearing and route he must 
traverse to make it to Checkpoint #4. Have subject return to Checkpoint 
#4. Mark route and any turn which leads the subject away from Checkpoint 
#4. Allow a maximum of ten minutes to return to Checkpoint #4) 

xii) Task 11. (Survey Knowledge) Wave subject arrange magnets on the white 
board indicating the location of the starting point and nine checkpoints. 
Measure time and note method of magnet placement (i.e. in order of visit, 
outside-in, or inside-out). Take picture of final results (allow 5 minutes 
maximum). 


g) Error (Definition) 


Subject strays from designated route (5 meters from designated route on a 
path/trail/road; 15 meters from cross country designated route). (record one error) 


Debriefing. 


a) Time — 30 Min 

b) Location — Graphics Lab 

c) OIC — CPT Quay B. Jones 

d) Materials — Clipboard with subject’s map & designated route, Data Collection 
Sheet, red pen to record data, GPS system, Troop (PC) w/ Arcview and Fort Ord 
Maps, digital camera, Participant Questionnaire(s), Researcher’s Script 

e) Administer questionnaire(s); down load GPS datum and display on aerial photo 
using Arcview. 

f) Discuss route. 


1) Have the subject complete the Debriefing Questionnaire. Read their answers 
and ask for any clarification. 

ii) Walk the subject through his route using the subjects planned route and the 
GPS data down loaded from the Message Pad and plotted on the aerial 
photo in Arcview. 


(a) Have the subject to explain why they deviated from their route at those 
locations where the two differ. 

(b) Have the subject explain when &. how they determined they were off 
course. 

(c) Have the subject explain how they recovered. 


iil) Ask the subject if he would have done anything different in the training phase 
now that has completed the experiment. 

iv) How much time does the subject spend playing computer games or working 
with computer graphics (more than an hour a day, a couple hours a week, 
once or twice a month, rarely, never)? 
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APPENDIX B. TASK LISTING 


Task 1. (Path Knowledge) Move from starting point to Checkpoint #1 along 
designated route (measure elapsed time and number of errors; mark deviation from route on 
map). 

Task 2. (Path Knowledge) Move from Checkpoint #1 to Checkpoint #2 along 
designated route (measure elapsed time and number of errors; mark deviation from route on 
map). 

Task 3.1. (Survey Knowledge) Take bearings to SP, CP #5, and CP #9 at the 
south side of CP #4). 

Task 3.2. (Path Knowledge) Move from Checkpoint #2 to Checkpoint #3 along 
designated route (measure elapsed time and number of errors; mark deviation from route on 
map). 

Task 4. (Path Knowledge) Move from Checkpoint #3 to Checkpoint #4 along 
designated route (measure elapsed time and number of errors; mark deviation from route on 
map). 

Task 5.1. (Survey Knowledge) Take bearings to CP #1, CP #6, and CP #8 at the 
south side of CP #4). 

Task 5.2. (Path Knowledge) Move from Checkpoint #4 to Checkpoint #5 along 
designated route (measure elapsed time and # errors; mark deviation from route on map). 

Task 6. (Path Knowledge) Move from Checkpoint #5 to Checkpoint #6 along 
designated route (measure elapsed time and number of errors; mark deviation from route on 
map). 

Task 7. (Path Knowledge) Move from Checkpoint #6 to Checkpoint #7 along 
designated route (measure elapsed time and number of errors; mark deviation from route on 
map). 

Task 8. (Path Knowledge) Move from Checkpoint #7 to Checkpoint #8 along 
designated route (measure elapsed time and number of errors; mark deviation from route on 


map). 
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Task 9. (Path Knowledge) Move from Checkpoint #8 to Checkpoint #9 along 
designated route (measure elapsed time and number of errors; mark deviation from route on 
map). 

Task 10. (Survey Knowledge) Have subject indicate bearing and route he must 
traverse to make it to Checkpoint #4. Have subject return to Checkpoint #4 (mark route 
and any turn which leads the subject away from Checkpoint #4. Allow a maximum of ten 
minutes to return to Checkpoint #4). 

Task 11. (Survey Knowledge) Have subject arrange magnets on the white board 
indicating the location of the starting point and nine checkpoints. Measure time and note 
method of magnet placement (i.e. in order of visit, outside in, or inside out). Take picture of 


final results (allow 5 minutes maximum). 
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APPENDIX C. BRIEFING SCRIPTS 
1. GENERAL 


The scripts in the appendix appear in the same format utilized for the experiment and do 
not follow the standard thesis format utilized in the chapters of this document. This appendix 
consists of five briefing scripts: In Briefing, Low Fidelity Map Group Briefing, Train-To 
Standard Map Group Briefing, Low Fidelity VE Briefing, Train-to-Standard VE Briefing, and 
the Course Briefing. Each participant receives the In Briefing and Course Briefing. The 
participants are exposed to either the Control Group Briefing, Map Group Briefing, or Virtual 
Environment Briefing depending on which group they are assigned. This appendix also contains 
the Debriefing hand out. 

2. IN BRIEFING 


Welcome to the Naval Postgraduate School's Computer Science Department. My name 
is CPT Quay B. Jones. Thank you for your assistance with today's experiment. Today's 
experiment deals with dismounted navigation in natural terrain. 


This experiment is not a test of your intelligence or performance. Rather, it is an 
evaluation of navigational tools. (For Military Personnel) Your performance will not be 
recorded in your personnel records but is intended for research purposes only. All information 
collected is for academic research only. Prior to starting the experiment you will be asked to 
read and sign a series of consent forms. Upon signing the consent forms, you will take self- 
evaluation, map reading, and spatial orientation exams. After the tests, you will under go a 
sixty-minute train-up period prior to moving to the navigation course. Upon completing the 
course, you will be brought back to Spanagel Hall for a short debriefing. 


If there are no questions, please read and sign this consent form. 
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3. LOW FIDELITY MAP GROUP BRIEFING 


In front of you is a 1:24,000 scale map of an orienteering course. On the flip side 
of the map is the same map enlarged to 1:5,000 scale but with no additional information. 
You have an aerial photograph or the area. You also have a clue sheet describing the 
location of the control points as well as photos of the control points. The map and photos 
are for your use to study and plan the route you will be using to navigate the course. 


You have sixty minutes to study the map. Your planned route must navigate you 
through the nine checkpoints in order. (Show the participant the checkpoints in order 
then point out each checkpoint in the photo.) Beginning at the designated starting point, 
you will go to CP1, then to CP2, then to CP3, ... and finally to CP9. The checkpoints are 
described in the clue sheet provided. You may take the clue sheet with you when you go 
on the course. Before the end of the sixty-minute study phase, you will mark your planned 
route on the aerial photograph using a red alcohol marker. 


After completing the study phase, you will be taken to the navigation course to run 
the route you designated on your aerial photograph. While navigating the course, you will 
not have the map nor will you be allowed to use a compass. During the execution of the 
course, you may request a one-minute map or a 30-second compass check; or a ninety- 
second map and compass check. You can request as many map or compass checks as you 
wish, but each check will be recorded. If you decide to deviate from your previously 
planned route, you may request the map to mark your newly planned route. 


Do you have any questions before we begin? 
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4. TRAIN-TO-STANDARD MAP GROUP BRIEFING 


In front of you is a map of an orienteering course. You also have a clue sheet 
describing the location of the control points as well as photos and screen capture images 
of the control points. The map, photos, and clue sheet are for your use to study and plan 
the route you will be using to navigate the course. 


You will now study the map and photos to plan and memorize your route. Your 
planned route must navigate you through the nine checkpoints in order. (Show the subject 
the checkpoints in order.) Beginning at the designated starting point, you will go to CP1, 
then to CP2, then to CP3, ... and finally to CP9. The checkpoints are described in the clue 
sheet provided. You may take the clue sheet with you when you go on the course. You 
will mark your planned route on the map using a red alcohol marker. 


Once you have finished planning your route, you should study it in order to 
memorize it. When you feel you have memorized it, (previous subjects have taken up to an 
hour to plan and study their route) you will be tested prior to departing for the course. If 
you do not adequately pass the test, you will again be given time to study your route. You 
must pass the pre-course test to standard before being allowed on the course. 


The pre-course test will consist of you verbally explaining your route from 
memory using only the clue sheet provided. You will explain how you are moving 
between the control points, and what clues you plan to use to assist you. During this test 
you will be allowed to view the map twice in order to refresh your memory but only for 30 
seconds each. Be as detailed and descriptive as possible. 


After completing the study phase, you will run the route you designated on your 
laminated map. While navigating the course, you will not have the map nor will you be 
allowed to use a compass. During the execution of the course, you may request a thirty 
seconds map or compass check; or a sixty-second map and compass check. You can 
request as many map or compass checks as you wish, but each check will be recorded. If 
you decide to deviate from your previously planned route, you may request the map to 
mark your newly planned route. 


Do you have any questions before we begin? 
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5. TRAIN-TO-STANDARD VE GROUP BRIEFING 


Prior to beginning the study phase you will under go a fifteen-minute model 
familiarization phase. This is to help you become accounted with the model controls prior 
to starting the experiment. The model you will be using for this phase bares no 
resemblance to the actual model to be used during the training phase. You will be required 
to show proficiency with the interface prior to moving on to the terrain model. 


In front of you is the 3-screen configuration, a joystick interface, and a list of 
instruction for the use of the interface (demo controls). Please feel free to explore the 
environment and control for the next few minutes. When you feel confident with the 
controls, I will walk you through a serious of questions to demonstrate your expertise. If 
you feel you need more time then please say so. 


(Conduct Familiarization Phase; after demonstrating their proficiency with the 
interface, load up the terrain model and begin the training phase) 


In front of you is a map of an orienteering course as well as a high fidelity 3-D 
model of the terrain depicted on the map. You also have a clue sheet describing the 
location of the control points as well as photos and screen capture images of the control 
points. The map, photos, and VE are for your use to study and plan the route you will be 
using to navigate the course. 


You will now study the map and VE to plan and memorize your route. Your 
planned route must navigate you through the nine checkpoints in order. (Show the subject 
the checkpoints in order, paging through each page as you go to each checkpoint.) 
Beginning at the designated starting point, you will go to CP1, then to CP2, then to CP3, 
... and finally to CP9. The checkpoints are described in the clue sheet provided. You may 
take the clue sheet with you when you go on the course. You should utilize both the map 
and VE in your planning. Once you have decided upon a route, you will mark your 
planned route on the map using a red alcohol marker. 


Once you have finished planning your route, you should use the map and VE to 
assist you in memorizing it. Once you feel you have memorized your route (previous 
subjects have taken up to an hour or more to plan and study their route) you will be tested 
prior to going to the course. If you do not pass the pre-course test, you will not be taken 
to the course, but will instead be given more time to study your route. You must pass the 
pre-course test to standard before being allowed on the real course. 


The pre-course test will consist of you being placed at the start point in the VE and 
running the course while following your planned route. You will not be allowed to utilize 
the 15 meter “pop-up” view during this test. You will not be allowed to utilize the “you 
are here” map during this test. You are allowed to take up to two map checks using the 
actual laminated map of the area. You are allowed to make compass checks as needed, but 
cannot run the course with the compass “always on”. If you become lost, leave your 
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planned route for 5 minutes, leave the boundaries or encounter a check point out of order 
I will stop you, perform a mandatory map check, record a mistake, and allow you to 
continue. On the third mistake I will conclude the test and give you more time to study. 
Note that conducting a search for a nearby point does not count as being lost. 


After completing the pre-course test, you will be taken to the navigation course to 
run the route you designated on your laminated map. While navigating the course, you will 
not have the map nor will you be allowed to use a compass. During the execution of the 
course, you may request a thirty seconds map or compass check; or a sixty-second map 
and compass check. You can request as many map or compass checks as you wish, but 
each check will be recorded. If you decide to deviate from your previously planned route, 
you may request the map to mark your newly planned route. 


Do you have any questions before we begin? 
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6. LOW FIDELITY MAP VE GROUP BRIEFING 


Prior to beginning the study phase you will under go a fifteen-minute model 
familiarization phase. This is to help you become comfortable with the model controls 
prior to starting the experiment. The model you will be using for this phase bears no 
resemblance to the actual model to be used during the training phase. You will be required 
to show proficiency with the interface prior to moving on to the terrain model. 


In front of you are the 3-screen configuration, a joystick interface, and a list of 
instructions for the use of the interface (demo controls). Please feel free to explore the 
environment and controls for the next few minutes. When you feel confident with the 
controls, I will walk you through a serious of questions to demonstrate your expertise. 


(Conduct Familiarization Phase; after the participant demonstrates proficiency 
with the interface, load up the terrain model and begin the training phase) 


In front of you is a 1:24,000 scale map of an orienteering course as well as a high 
fidelity 3-D model of the terrain depicted on the map. On the flip side of the map is the 
same map enlarged to 1:5,000 scale. No additional information is included on the enlarged 
map. Along with the map, you have an aerial photograph of the region. You also have a 
clue sheet describing the location of the control points as well as photos and screen 
capture images of the control points. The maps, photos, and VE are for your use to study 
and plan the route you will be using to navigate the course. 


You have sixty minutes to study the map and VE. Your planned route must 
navigate you through the nine checkpoints in order. (Show the participant the 
checkpoints in order then point out each checkpoint in the photo.) Beginning at the 
designated starting point, you will go to CP1, then to CP2, then to CP3, ... and finally to 
CP9. The checkpoints are described in the clue sheet provided. You may take the clue 
sheet with you when you go on the course. Before the end of the sixty-minute study 
phase, you will mark your planned route on the aerial photograph using a red alcohol 
marker. 


After completing the training phase, you will be taken to the navigation course to 
run the route you designated on your aerial photograph. While navigating the course, you 
will not have the map nor will you be allowed to use a compass. During the execution of 
the course, you may request a one-minute map or a 30-second compass check; or a 90- 
second map and compass check. You can request as many map or compass checks as you 
wish, but each check will be recorded. If you decide to deviate from your previously 
planned route, you may request the map to mark your newly planned route. 


Do you have any questions before we begin? 
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7. COURSE BRIEFING 

Pick-up participant from the Graphics Lab. 

Move participant to the Fort Ord orienteering course. 
Move participant to start point: 

Brief the participant on animals and ammunition 


“You are at the start point of the Navigation Course. During the experiment, I may stop 
you and ask you to answer questions. You must navigate the nine checkpoints in order. 
Each control point will be identified by a control point marker (show participant a control 
marker) which you must touch prior to moving to the next control point. Once you touch 
a control marker, I will tell you which marker it is. If it is the correct marker, I will give 
you further instructions. If it is the incorrect marker, I will not say anything other then the 
marker’s number. I will not stop you unless you attempt to cross the course boundaries 
(show participant the boundaries). You may request the compass for a thirty second 
compass check; the map/maps for a (30/60) second map check; or the map and compass 
for a (60/90) second compass and map check. These checks will be recorded and timed by 
me. If you determine that you would like to change your route, you may request the map 
and a blue marker to mark changes to your proposed route. You will have sixty seconds 
to mark your new route. You may request an additional sixty seconds if you deem it 
necessary. You have sixty minutes to make it as far as you can along your planned route. 
From now until completion of the navigation course do not interact with anyone. Before 
you begin, do you have any questions?” 


TASK 1: START POINT TO CHECKPOINT ONE. 


Task: “Your first task is to move from the start point to checkpoint one along your 
designated route." 


Condition: “Without a map or interaction with anyone move from Start point to 
checkpoint one along your preplanned route. If you deviate from the designated route you 
will be allowed to continue your movement unless you attempt to go outside the course 
boundaries. You may deviate 5m from your route, if you are on a trail, or 15m, if you are 
conducting cross-country movement before you are assessed an error. You can move back 
and forth along your route without being assessed an error. If you deviate from your path 
for more then 15 continues minutes and are not make progress towards the intended 
control point, I will stop you, show you your location on the map, and give you sixty 
seconds to mark a new route to the appropriate control point.” 


Standard: “Do the best you can.” 


“Ready,... Begin.” 
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TASK 2: CHECKPOINT ONE TO CHECKPOINT TWO. 


Task: “Checkpoint one. Your next task is to move from the checkpoint one to checkpoint 
two along your planned route. Conditions and standards are unchanged.” 


TASK 3.1.A, B, C: SPATIAL AWARENESS TEST I. 


Stop timer 
Stop participant at spatial awareness test area. 


“Checkpoint two. Stop, I am going to have you identify the direction to three 
checkpoints.” 


Place the color wheel platform in its base on the south side of checkpoint. 

Task: “Identify the direction to the start point, checkpoint five, and checkpoint nine.” 
Show participant arrows as you state their names. 

Condition: “Given a color coded, 360-degree wheel and three arrows, identify the 
direction to the start point, checkpoint five, and checkpoint nine by placing the appropriate 
arrow in the direction of its checkpoint.” 

Standard: “Unchanged.” 

Record the time it takes the participant to perform the Wheel task and the 
orientation of the participant (looking north, south, east, rotates in the direction of the 
arrows, etc). Once done, photo graph the wheel, remove wheel platform from its stand, 
and have participant continue to checkpoint three. 


TASK 3.2: CHECKPOINT TWO TO CHECKPOINT THREE. 


Task: “Your next task is to move from the checkpoint two to checkpoint three along your 
planned route. Conditions and standards are unchanged. Ready,... Begin.” 


Start timer 
TASK 4: CHECKPOINT FOUR TO CHECKPOINT FIVE. 


Task: “Checkpoint three. Your next task is to move from the checkpoint three to 
checkpoint four along your planned route. Conditions and standards are unchanged.” 
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TASK 5.1.A, B, C: SPATIAL AWARENESS TEST I. 
Stop timer 
Stop participant at spatial awareness test area. 


“Checkpoint four. Stop, I am going to have you identify the direction to three 
checkpoints.” 


Place the color wheel platform in its base on the south side of checkpoint. 

Task: “Identify the direction to checkpoint one, checkpoint six, and checkpoint eight.” 
Show participant arrows as you state their names. 

Condition: “Given a color coded, 360-degree wheel and three arrows, identify the 
direction to checkpoints one, six, and eight by placing the appropriate arrow in the 
direction of its checkpoint.” 

Standard: “Unchanged.” 

Record the time it takes the participant to perform the Wheel task and the orientation 
of the participant (looking north, south, east, rotates in the direction of the arrows, 
etc). Once done, photo graph the wheel, remove wheel platform from its stand, and 
have participant continue to checkpoint five. 


TASK 5.2: CHECKPOINT TWO TO CHECKPOINT THREE. 


Task: “Your next task is to move from the checkpoint four to checkpoint five along your 
planned route. Conditions and standards are unchanged. Ready,... Begin.” 


Start timer 


TASK 6: CHECKPOINT FIVE TO CHECKPOINT SIX. 


Task: “Checkpoint five. Your next task is to move from the checkpoint five to 
checkpoint six along your planned route. Conditions and standards are unchanged.” 


TASK 7: CHECKPOINT SIX TO CHECKPOINT SEVEN. 


Task: “Checkpoint six. Your next task is to move from the checkpoint six to checkpoint 
seven along your planned route. Conditions and standards are unchanged.” 
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TASK 8: CHECKPOINT SEVEN TO CHECKPOINT EIGHT. 


Task: “Checkpoint seven. Your next task is to move from the checkpoint seven to 
checkpoint eight along your planned route. Conditions and standards are unchanged.” 


TASK 9: CHECKPOINT EIGHT TO CHECKPOINT NINE. 


Task: “Checkpoint eight. Your next task is to move from the checkpoint eight to 
checkpoint nine along your planned route. Conditions and standards are unchanged.” 


TASK 10.1: CHECKPOINT 4 IDENTIFICATION. 


While standing at checkpoint nine: 


Stop timer 


Task: “Checkpoint nine, finish point. Your next task is to identify the location of 
checkpoint four from where you are.” 


Condition: “Point to checkpoint four and tell me where checkpoint four is from here. 
(i.e., twenty meters and in this direction).” 


Standard: “Unchanged.” 


TASK 10.2: DESCRIBE ROUTE FROM CHECKPOINT NINE TO START 
POINT 


Task: “Your next task is to describe what you consider the easiest route you would take 
to move from here to checkpoint four.” 


Condition: “Without a map, describe the route you would take to move from checkpoint 
nine to checkpoint four.” 


Standard: “Unchanged.” 


TASK 10.3: CHECKPOINT NINE TO START POINT (if described route would 
take them in the general location of the start point) 


Task: “Your next task is to move from checkpoint nine to checkpoint four using the route 
you just described.” 


Condition: “Again, do not interact with anyone to include the researcher. You may not 
request a map or a compass check." 


Standard: "You have ten minutes, otherwise standards are unchanged." 
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"Ready,... Begin" 
Start timer 


Reach checkpoint #4 or ten minutes has elapsed. 


FINISH 


Stop timer 


“Stop. Congratulations you have completed the navigation portion of this experiment. 
We will now return to the vehicle for one final test before returning to the laboratory.” 


TASK 11: WHITE BOARD TEST. 


Task: “Your final task is to create a top down representation of the start point and nine 
control points.” 


Condition: “Without a map or interaction with anyone take the ten magnets labeled with 
the start point and nine checkpoints (show the participant the magnets) and place them 
on a clean white board in proper perspective to each other. You are attempting to create 
a top down view of the checkpoints, actual distance between points does not matter, 
however, relative locations to each checkpoint does. Until you feel you are finished or 
five minutes has elapsed, you may place and move the magnets as you wish. 

Standard: “Do the best you can.” 


"Any questions,... Ready,... Begin." 


Start timer 
Stop the timer when the participant indicates he has finished or ten minutes has 
elapsed, which ever occurs first. Observe the participant and note his method for 


placing the magnets (i.e. in order of visit, outside in, or inside out). Take a picture of 
the final results (allow participant 5 minutes maximum to perform the task). 


Stop timer 


"Stop. Congratulations on completing the final task for this experiment. We will now 
return to NPS for a final debriefing session." 


Move participant back to the Graphics Lab for debriefing. 
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8. DEBRIEFING 


The use of virtual environments in training and education has been an expanding 
field for the last two decades. With recent developments in computer systems, virtual 
reality models are now able to display much higher fidelity. In order to insure we are 
providing a positive training transfer and properly replicating real world environments, 
research is being conducted in the levels of detail required in models. 


The study you have just completed is concerned with gathering information on 
how individuals navigate through complex virtual environments. You spent a session 
planning and studying a route demonstrating route knowledge Finally, you demonstrated 
spatial knowledge of the terrain through estimating bearings to known points and 
movement to an unplanned location. 


Four separate groups were examined in order to determine performance levels. All 
four groups were given a map on which they designated their routes prior to running the 
navigation course. The first group was only allowed to study a 1:24,000 map for 60 
minutes. The second group was given the 1:24,000 map and allowed to maneuver through 
a real time, high fidelity virtual representation of the terrain for 60 minutes. The third 
group had only a 1:5,000 scale map but had to train to a set standard before being allowed 
on the actual outdoor course. The fourth group had the 1:5,000 scale map and the VE and 
also had to complete the training to a standard before running the course. 


The research personnel observed and recorded information based on the 
experience and behavior of the participants in order to gather the information equipped for 
the redesign and implementation of a more useful virtual model The notes and 
observations collected will be used for the purpose of establishing standards for model 
development. 


Your assistance in this project will contribute to the production of more useful 
virtual environments that provide users with spatial knowledge and better navigational 
skills. With the information gathered from your experience and the experience of other 
participants, we are discovering what people generally use as navigational cues in the 
virtual and real world environments. This information will assist in the design of future 
virtual reality models that will be adaptive to a variety of individual needs. 


If you have any questions about the study, please ask your research assistant. 


Until 30 August 1999, please do not discuss this experiment with anyone except our 
research personnel to prevent influencing any future participants. Thank you for your 


participation in this study. 


The research supervisor, CPT Quay B. Jones, for this study can be contacted at 
(408) 656 — 4077 or Email: jones 9cs.nps.navy.mil. 


APPENDIX D. CONSENT FORMS 


1. GENERAL 

The forms in the appendix appear in the same format utilized for the experiment and do 
not follow the standard thesis format utilized in the chapters of this document. This appendix 
consists of three documents: Consent Form, Minimal Risk Consent Statement, and the Privacy 
Act Statement. Each participant is required to read and sign these documents before he is 
allowed to participate in the study. A research monitor observes and verifies the signing of 
each document. The format and content of these documents is based on the forms used in 
MAJ Wiliam Banker’s land navigation experiment [BANK 97] and MAJ Simon R. Goerger’s 
land navigation experiment [GOER 98b]. 


125 


. CONSENT FORM 


PARTICIPANT CONSENT FORM 


. Introduction. You are invited to participate in a study of spatial awareness of natural 
and virtual environments. With information gathered from you and other participants, 
we hope to discover insight on navigational aids used to move through virtual 
environments during dismounted navigation of natural terrain. We ask you to read and 
sign this form indicating that you agree to be in the study. Please ask any questions you 
may have before signing. 


. Background Information. The Naval Postgraduate School NPSNET Research Group 
is conducting this study. 


. Procedures. If you agree to participate in this study, the researcher will explain the 
tasks in detail. There will be two sessions: a) 30 pretest phase and 2) training and 
execution phases lasting approximately five hours in duration, during which you will be 
expected to accomplish a number of tasks related to navigating natural terrain. 


. Risks and Benefits. This research involves no risks or discomforts greater then those 


encountered in ordinary hike through rolling, wooded terrain. The benefits to the 
participants are gaining techniques for enhancing spatial knowledge of unfamiliar 
environments and contributing to current research in human-computer interaction. 


Compensation. No tangible reward will be given. A copy of the results will be 
available to you at the conclusion of the experiment. 


. Confidentiality. The records of this study will be kept confidential. No information 
will be publicly accessible which will possibly identify you as a participant. 


. Voluntary Nature of the Study. If you agree to participate, you are free to withdraw 
from the study at any time without prejudice. You will be provided a copy of this form 
for your records. 


. Points of Contact. If you have any further questions or comments after the completion 
of the study, you may contact the research supervisor, CPT Quay B. Jones, at (408) 
656 — 4077 (Email: jones @cs.nps.navy.mil). 


. Statement of Consent. I have read the above information. I have asked all question 
and have had my questions answered. I agree to participate in this study. 


Researcher's Signature Date 
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3. MINIMAL RISK CONSENT STATEMENT 


NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 
MINIMAL RISK CONSENT STATEMENT 


Participant: VOLUNTARY CONSENT TO BE A RESEARCH PARTICIPANT IN: Virtual 
Environments and Navigation in Natural Environments 


D 


I have read, understand and been provided "Information for Participants" that provides the 
details of the below acknowledgments. 


I understand that this project involves research. An explanation of the purposes of the 
research, a description of procedures to be used, identification of experimental procedures, 
and the extended duration of my participation have been provided to me. 


I understand that this project does not involve more than minimal risk. I have been informed 
of any reasonably foreseeable risks or discomforts to me. 


I have been informed of any benefits to me or to others that may reasonably be expected from 
the research. 


I have signed a statement describing the extent to which confidentiality of records identifying 
me will be maintained. 


I have been informed of any compensation and/or medical treatments available if injury 
occurs and is so, what they consist of, or where further information may be obtained. 


I understand that my participation in this project is voluntary, refusal to participate will 
involve no penalty or loss of benefits to which I am otherwise entitled. I also understand that 
I may discontinue participation at any time without penalty or loss of benefits to which I am 
otherwise entitled. 


I understand that the individual to contact should I need answers to pertinent questions about 
the research is Rudy Darken, Ph.D., Principal Investigator, and about my rights as a research 
participant or concerning a research related injury is the Modeling Virtual Environments and 
Simulations Chairman. A full and responsive discussion of the elements of this project and 
my consent has taken place. 


Medical Monitor: Flight Surgeon, Naval Postgraduate School 


Signature of Principal Investigator Date 
Signature of Volunteer Date 
Signature of Witness Date 
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4. PRIVACY ACT STATMENT 


NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 
PRIVACY ACT STATEMENT 


1. Authority: Naval Instruction 


2. Purpose: Spatial Cognition information will be collected to enhance knowledge, or 
to develop tests, procedures, and equipment to improve the development of Virtual 
Environments. 


3. Use: Spatial Cognition information will be used for statistical analysis by the 
Departments of the Navy and Defense, and other U.S. Government agencies, 
provided this use is compatible with the purpose for which the information was 
collected. Use of the information may be granted to legitimate non-government 
agencies or individuals by the Naval Postgraduate School in accordance with the 
provisions of the Freedom of Information Act. 


4. Disclosure/Confidentiality: 


a. I have been assured that my privacy will be safeguarded. I will be assigned a 
control or code number which thereafter will be the only identifying entry on 
any of the research records. The Principal Investigator will maintain the cross- 
reference between name and control number. It will be decoded only when 
beneficial to me or if some circumstances, which is not apparent at this time, 
would make it clear that decoding would enhance the value of the research data. 
In all cases, the provisions of the Privacy Act Statement will be honored. 


b. I understand that a record of the information contained in this Consent Statement 
or derived from the experiment described herein will be retained permanently at 
the Naval Postgraduate School or by higher authority. I voluntarily agree to its 
disclosure to agencies or individuals indicated in paragraph 3 and I have been 
informed that failure to agree to such disclosure may negate the purpose for 
which the experiment was conducted. 


c. I also understand that disclosure of the requested information, including my 
Social Security Number, is voluntary. 


Signature of Volunteer Name, Grade/Rank (if applicable) DOB SSN Date 


Signature of Witness Date 
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APPENDIX E. QUESTIONNAIRES AND TESTS 
1. GENERAL 


The items in the appendix appear in the same format utilized for the experiment 
and thus do not follow the standard thesis format utilized in the chapters of this 
document. This appendix consists of eight documents: Land Navigation Questionnaire, 
Self Ability Evaluation, Santa Barbara Sense-of-Direction Scale, Map Reading Test, 
Guilford-Zimmerman Aptitude Survey (cover only), Practice Model Test, and two 
Debriefing Questionnaires. This tests and questionnaires are the same as in Goerger’s 
thesis [GOER 98b]. 

The Land Navigation Questionnaire (Appendix E.2) gathers general navigational 
background of the participant. The participant completes this prior to beginning the 
experiment. 

The Self Ability Evaluation (Appendix E.3) is a qualitative self analysis of an 
individual’s navigational ability. It provides a participant with general limits from which 
to appraise his perceived navigation aptitude. The left end of the scale is valued at 0.00 
and the right end of the bar line is valued at 1.00. Values measured from 0.00 to 0.33 are 
assessed as beginning navigators. From 0.33 to 0.66 is ranked as an intermediate 
navigator. Values of 0.66 to 1.00 are evaluated as experts. This test was included to 
negate any bias inherent in the word "beginner" from the Land Navigation Questionnaire. 

The Santa Barbara Sense-of-Direction Scale (Appendix E.4) is a quantitative self- 
evaluation of navigational ability. The University of California at Santa Barbara 
developed the scale. An individual's score is calculated by reversing the values of 
questions 2, 6, 8, 10, 11, 12, 13, and 15. For example, if the participant answered 
question number two as “3”, the question is given a numerical value of "5". Once the 
values for the above questions are reversed, sum the value of each question and divide the 
total by the number of questions answered. The lower the resulting score the more 
confidant an individual is in their navigational abilities. The test scale has a mean score of 
3.54 with a standard deviation of 1.03. For this experiment, most individuals expressed 
high confidence in their navigational abilities. 

The Map Reading Test (Appendix E.5) consists of twenty questions dealing with 


terrain identification. The test is designed to determine an individual's proficiency in 
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reading the terrain features on a map and associating them to real world terrain features. 
The first fifteen questions concern properly identifying terrain features from 1:50,000 
scale military maps. The final five questions dealt with matching images of terrain 
features to map depictions of terrain features. The answers for the test are listed in Table 
E.1. Each question is worth one point. If a participant misidentifies a linear terrain feature 
they receive 0.5 points for the question. For example, if the terrain feature is a stream and 
the participant classifies it as a road, they receive 0.5 points for the question. However, if 
the participant describes a stream as a draw, they receive no credit. Participants must 


score 65% (13 out of 20) or better to be allowed to participate in the study. 


Dek? cee e 

Le aee mmc 

th ue M 
1.5 Hill Top 


з p meme 
ван 


ы. ЖЕСТ Road/Trail 


Table E.1 Map Test Answer Key 


Spur/Finger 
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The Guilford-Zimmerman Aptitude Survey (Appendix E.6) evaluates an 
individual's spatial orientation ability. The results of this test are compared to a pool of 
national test scores to determine if a participant is above or below the national average 
for spatial orientation. Those above the national average are describe as having a "high" 
GZ score, or high spatial ability. Those scoring below the average are classified as “low”. 
The experiment groups were set up to evenly distribute high and low ability subjects 
evenly between the groups. 

The Practice Model Test (Appendix E.7) is administered to each VE participant 
prior to moving onto the actual course model. It is used to provide the participant with 
familiarity with the interface functions. Each virtual environment participant was 
required to complete each task of the Practice Model Test. After completing the test, a 
participant is retested on any functions they failed to properly employ until he is able to 
do so. 

The Debriefing Questionnaires (Appendices E.8 and E.9) are administered upon 
return from running the course and prior to the final review of the participant’s route. 
Participants in the Map Only Groups received the questionnaire in Appendix E.8. Virtual 
Environment participants receive the questionnaire in Appendix E.9 that has an additional 
page containing questions related to the VE and its interface. The questions are designed 
to provide a qualitative analysis of the training materials and course [GOER 98b]. A five 
point scale (1-5) is used for the questionnaire. The last page of the questionnaire is used 
to determine what details subjects think are necessary for inclusion in a useful VE. 
Goerger deliberately left stream/rivers off of the list to see how much attention subjects 
were paying. 


The raw scores from these tests and questionnaires are listed in Appendix O. 
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2. LAND NAVIGATION QUESTIONNAIRE 


Name: Age: Sex: 


Branch of Service: Rank: 


1) Where did you first learn to navigate? 


a) Scouting, Boys/Girls Club 
b) Parents 

c) Friend 

d) ROTC/Academy 

e) Basic Training 

f) Officer Candidate School 
g) Officers Basic Course 

h) Other: 


2) How many years have you been Orienteering/Navigating? 


a) less then a year 

b) one year or more 
с) two years or more 
d) five years or more 
e) ten years or more 


3) At what level would you classify your navigating abilities? 
a) Novice/Beginner 
b) Intermediate/Average 
c) Expert/Advanced 
4) How many Land navigation or Orienteering courses have you done in the last year? 
5) The land navigation course runs through varying degrees of vegetation and over 
rolling terrain. It will require you to negotiate a distance of no more than three miles in 


one hour. Do you have any physical disabilities that would prevent you from executing 
this task? Yes/No 
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3. SELF ABILITY EVALUATION 


Participant ID: 


The following bar line depicts the navigation ability evaluation of an average 
infantry officer with five years experience. The "X" indicates his ability level. 


| 
Knows how to Navigates with 
read a map no errors; 
Rarely looks 
at map 


Place an “X” on the line below were you feel your navigational abilities are at this 
time. 


—— 


Knows how to Navigates with 
read a map no errors; 
Rarely looks 
at map 
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4. SANTA BARBARA SENSE-OF-DIRECTION SCALE 
Participant ID: Date: SEX: F M AGE: 


This questionnaire consists of several statements about your spatial and 
navigational abilities, preferences, and experience. After each statement, you should 
circle a number to indicate your level of agreement with the statement. Circle “1” if you 
strongly agree that the statement applies to you, “7” if you strongly disagree, or some 
number in between if your agreement is intermediate. Circle “4” if you neither agree nor 
disagree. 

1. Iam very good at directions. 

strongly agree 1 2 3 4 5 6 7 strongly disagree 
2. Ihave a poor memory for were I left things. 

strongly agree 1 2 3 4 5 6 7 strongly disagree 
3. Iam very good at judging distances. 

strongly agree 1 2 3 4 5 6 7 strongly disagree 
4. My "sense of direction" is very good. 

strongly agree 1 2 3 4 5 6 7 strongly disagree 
5. I tend to think of my environment in terms of cardinal directions (N, S, E, W) 

strongly agree 1 2 3 4 5 6 7 strongly disagree 
6. Ivery easily get lost in a new city. 

strongly agree 1 2 3 4 5 6 7 strongly disagree 
7. I enjoy reading maps. 

strongly agree 1 2 3 4 5 6 7 strongly agpi 
8. Ihave trouble understanding directions. 


strongly agree 1 2 3 4 5 6 7 strongly disagree 


(turn over and continue) 
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10. 


ШЕ 


12. 


13. 


14. 


15. 


I am very good at reading maps. 

strongly agree 1 2 3 4 5 6 7 strongly disagree 

I don't remember routes very well while riding as a passenger in a car. 
strongly agree 1 2 3 4 5 6 7 strongly disagree 

I don't enjoy giving directions. 

strongly agree 1 2 3 4 5 6 7 strongly disagree 

It's not important to me to know where I am. 

strongly agree 1 2 3 4 5 6 7 strongly disagree 

I usually let someone else do the navigational planning for long trips. 
strongly agree 1 2 3 4 5 6 7 strongly disagree 

I can usually remember a new route after I have traveled it only once. 
strongly agree 1 2 3 4 5 6 7 strongly disagree 

I don't have a very good “mental map” of my environment. 


strongly agree 1 2 3 4 5 6 7 strongly disagree 
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5. MAP READING TEST 


The following is a list of terrain features commonly found on military and/or 
orienteering maps. Using the list of terrain features, identify the most predominate 
terrain feature within each circle and place your answer in the space provided. Each 
terrain feature from the list may be used more than once or not at all. 


. Depression 

Draw 

Hill Top 

. Lake/Pond 

Ridge Line 

Road/Trail 

. Road/Trail Intersection 


Saddle 

Spur/Finger 
Stream/River 

. Stream/River Intersection 
Valley 


PASSE Osho oe > 
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Depression 

Draw 

Hill Top 

Lake/Pond 

Ridge Line 

Road/Trail 

Road/Trail Intersection 
Saddle 

Spur/Finger 
Stream/River 
Stream/River Intersection 
Valley 


. 
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Spur/Finger 


J. Stream/River 
K. Stream/River Intersection 


G. Road/Trail Intersection 
L. Valley 


E. Ridge Line 
F. Road/Trail 
H. Saddle 


A. Depression 


B. Draw 
D. Lake/Pond 


C. Hill Top 
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Using the following map representations, choose the best representation for each 
picture displayed below. The map representations are a facsimile of the terrain shown in 
the photos. Some map representations may be used more than once or not at all. 
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6. GUILFORD-ZIMMERMAN APTITUDE SURVEY 


The Guilford-Zimmerman Aptitude Survey 


a 


Part 5/Spatial Orientation Cai 19 eS Ce Boy c4 


AN ұу reseed. Not 19 be reproduced in woke Г? 
СЕЛ СУ тел руда о ве обим 
U^ Discretes by Cossoony Psychoingsis Press, te. 


2. 
Tm 

24 
і 


ізі 8,4 2 5, • 
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` Y. 
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INSTRUCTIONS. 
This is a test of your ability to see changes in direction and position. In each item you are to note how the 
position of the boat has changed in the second picture from the original position in the first picture. 


Here is Sample item 1. Ds These are the five possible answers to the item. 


These bars represent the boat's This is the prow (iront end) ol a 
prow. motor boat in which you are riding. 


This Is the aiming point. Itis the 
This is the correct answer. It shows ICT ain he Mu ca т "e 
that the prow of the boat has drop- AN qus signted nght over the po: 
ped below the aiming point. ot ine prow. 


{tf the prow had risen, instead of ; ^. 3 X This is the same aiming point 
dropped, the correct answer would : E. shown above. Note that the prow 
have been C, instead ot D.) „НН has dropped below it. 


Sample item 7 


To work each item: First, look al the top picture and see where the motor boat is headed. Second, look at the 
boltom picture and note the CHANGE in the boat's heading. Third, mark the answer that shows*the same change on 
the separate answer sheet. 


Try Sample Item 2. 


This atso shows that the prow of 

the boat ts to the right of the aiming : езе. 

point. So, itis the correct answer. 252-< This is the aiming point. 
=. IH * 


(If the boat had turned to the left, 
instead of to the right, the correct 


answer would have been A.) This is the same aiming point. 


The motor boat is now headed to 
the right of it. 


Sample Пет 2 


(> Consulting Psychologists Press, Inc., 3803 E. Bayshore Road, Palo Alto, CA 94303 
98 97 96 95 94 87654 





Figure E.1 Guilford-Zimmerman Aptitude Survey Cover Page 
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7. PRACTICE MODEL TEST 
a. Turn to a heading of 360 degrees and begin movement. 
b. Switch to atop down view 
c. Switch to a 15-meter view 
d. Change to run mode 
e. Change to walk mode 
f. Move to the road and take a right 


g. While following the road: 


i) Look-up 
ii) Look down 
Ш) Look left 


iv) Look Right 
h. Head into town 
i. Stop 
j. What is your heading? 
k. Begin movement. 
1 Run 
m. Slow down and stop at the road sign 
n. Look to your right. What do you see? 
o. Using the quick view keys, see what is at CP6 


p. Using the hot keys, return to the start point 
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8. DEBRIEFING QUESTIONNAIRES 


a. Map Group Debriefing Questionnaire 





Hard to Read Easy to Read 










Was the map easy to read? 1 2 3 4 5 N/A 
Hard to Understand Easy to Understand 
Was the map easy to understand? 1 2 У 4 5 N/A 
Were the trails & roads adequately Definitely Not Definitely Yes 
shown on the map? 1 2 3 4 5 N/A 
Were the man made structures Definitely Not Definitely Yes 
Adequately shown on the map? 1 2 3 4 5 N/A 
Were the obstacles adequately Definitely Not Definitely Yes 
shown on the map? 1 2 3 4 5 М/А 
Was the vegetation adequately Definitely Not Definitely Yes 
shown on the map? 1 2 3 4 5 М/А 
Using the map, how difficult was it Easy Very Difficult 
to plan your route? 1 2 3 4 5 N/A 





Comments: 





Easy Very Challenging 
How difficult was the course? 1 2 3 4 5 МА 
Definitely Not Definitely Yes 
Were the control points well marked? 1 2 3 4 5 N/A 
Were the control points located Definitely Not Definitely Yes 
where you expected them? ] 2 3 4 5 М/А 
Had routes been trampled down Definitely Not Definitely Yes 
leading to the control points? 1 2 3 4 5 М/А 
Did you have difficulties remembering Definitely Not Definitely Yes 
your planned route? 1 2 3 4 5 М/А 


Comments: 











Definitely Not Definitely Yes 
Did you enjoy this experiment? ] 2 3 4 5 N/A 
Did you feel the training phase Definitely Not Definitely Yes 
was long enough? ] 2 3 4 5 М/А 
Did you feel the training phase Definitely Not Definitely Yes 
was too short? ] 2 3 4 5 N/A 
Do you feel the training familiarized Definitely Not Definitely Yes 
you learn the environment? 1 2 3 4 5 М/А 
Did you feel confident in navigating Definitely Not Definitely Yes 
the terrain without a map or compass? 1 2 3 4 5 N/A 





Comments: 
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1. Place an "X" next to the items you feel must be replicated in a model that prepares you to navigate an 


actual piece of terrain. 


2. From the list of items in question # 1, choose and rank the six items you feel are the most important 
for a computer model which will be used to prepare an individual to navigate an actual piece of terrain. 


ол ы о бю — 


Buildings 
Buildings 
Buildings 
Buildings 
Buildings 


Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 


Terrain 
Terrain 
Terrain 
Terrain 
Terrain 
Terrain 
Terrain 


factory 

houses 

public buildings 
shacks 

other 


compass 
road signs 
rock piles 
sand bags 
Street signs 
the sun 
people 
animals 
sound 
other 
other 


clearings 
depressions 
hills 

knolls 
ridgelines 
spurs/fingers 
other 
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roads 
roads 
roads 
roads 
roads 


obstacles 
obstacles 
obstacles 
obstacles 
obstacles 
obstacles 
obstacles 


vegetation 
vegetation 
vegetation 
vegetation 
vegetation 


water 
water 
water 
water 
water 
water 


dirt roads 
foot paths 
paved roads 
trails 

other 


electric lines 
pits/fox holes 
shallow ditches 
telephone poles 
towers 
trenches 

other 


bushes 
flowers 
grass/weeds 
trees 

other 


lakes 

marsh lands 
ponds 
puddles 
swamps 
other 


b. Virtual Environment Group Debriefing Questionnaire 





Hard to Read Easy to Read 
Was the map easy to read? 1 2 3 4 5 N/A 
Hard to Understand Easy to Understand 
Was the map easy to understand? 1 2 3 4 5 N/A 
Were the trails & roads adequately Definitely Not Definitely Yes 











shown on the map? 1 2 3 4 5 N/A 
Were the man made structures Definitely Not Definitely Yes 
Adequately shown on the map? 1 2 3 4 5 N/A 
Were the obstacles adequately Definitely Not Definitely Yes 
shown on the map? 1 2 3 4 5 МА 
Was the vegetation adequately Definitely Not Definitely Yes 
shown on the map? 1 2 3 4 5 М/А 
Using the map, how difficult was it Easy Very Difficult 
to plan your route? 1 2 3 4 5 N/A 





Comments: 





Easy Very Challenging’ 
How difficult was the course? 1 2 3 4 5 М/А 
Definitely Not Definitely Yes 
Were the control points well marked? 1 2 2 4 5 М/А 
Were the control points located Definitely Not Definitely Yes 
where you expected them? 1 2 3 4 5 М/А 
Had routes been trampled down Definitely Not Definitely Yes 
leading to the control points? 1 2 3 4 5 М/А 
Did you have difficulties remembering Definitely Not Definitely Yes 
your planned route? 1 2 3 4 5 М/А 


Comments: 








MISC Definitely Not Definitely Yes 
Did you enjoy this experiment? 1 2 3 4 5 N/A 
Did you feel the training phase Definitely Not Definitely Yes 










was long enough? 1 2 3 4 5 N/A 
Did you feel the training phase Definitely Not Definitely Yes 

was too short? 1 2 3 4 5 М/А 
Do you feel the training familiarized Definitely Not Definitely Yes 

you learn the environment? 1 2 3 4 5 N/A 
Did you feel confident in navigating Definitely Not Definitely Yes 

the terrain without a map or compass? 1 2 3 4 9 N/A 
Comments: 
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MODEL Definitely Not Definitely Yes 
Was the model clear and viewable? 1 2 3 4 5 М/А 




















Definitely Not Definitely Yes 

Did the model coincide with the map? 1 2 2 4 5 М/А 
Were the trails & roads adequately Definitely Not Definitely Yes 
represented in the model? 1 2 3 4 5 М/А 
Were the man made structures Definitely Not Definitely Yes 
adequately represented in the model? 1 2 3 4 5 N/A 
Were the obstacles adequately Definitely Not Definitely Yes 
represented in the model? 1 2 3 4 5 М/А 
Was the vegetation adequately Definitely Not Definitely Yes 
represented in the model? 1 2 3 4 5 М/А 
Were changes in elevation adequately Definitely Not Definitely Yes 
represented in the model? 1 2 8 4 · 5 N/A 
Did the model help you identify the Definitely Not Definitely Yes 
control points within the last 50m? 1 2 3 4 5 N/A 
Did the model help you identify the Definitely Not Definitely Yes 
general area of the control points? 1 2 3 4 5 М/А 
Using the model, how difficult was it Easy Very Difficult 
to plan your route? 1 2 3 4 5 М/А 













Do you feel the model gave you an Definitely Not Definitely Yes 
advantage you normally wouldn't 1 2 3 4 5 М/А 
have had? 
Would you use this tool if it were Definitely Not Definitely Yes 
















available for mission planning? 1 2 3 4 5 М/А 
Would you use this tool if it were Definitely Not Definitely Yes | 
available for mission rehearsal? 1 2 3 4 5 М/А 









Would you use this tool if it were Definitely Yes 


available for navigation training? 


ODEL INTERFACE User Friendly 


Were you able to easily move through 1 2 3 4 5 М/А 
the model? 





























Confusing User Friendly 

Was the joystick easy to use? 1 2 3 4 5 М/А 
Confusing User Friendly 

Was the acceleration lever easy to use? 1 2 3 4 5 М/А 
Confusing User Friendly 

Were the toggle buttons easy to use? 1 2 3 4 5 N/A 
Confusing User Friendly 

Your overall felling about the interface? 1 2 3 4 5 М/А 
Was the 15-minute train-up on the Definitely Not Definitely Yes 
initial model useful? 1 2 3 4 5 N/A 










Was the 15-minute train-up on the Definitely Not Definitely Yes 
initial model enough time to become 1 2 3 4 5 N/A 
familiar with the interface? 
Did the use of three screens cause Definitely Not Definitely Yes 

any confusion when maneuvering? 1 2 3 4 5 М/А 


145 














1. Place an "X" next to the items you feel must be replicated in a model that prepares you to navigate an 


actual piece of terrain. 


2. From the list of items in question # 1, choose and rank the six items you feel are the most important 
for a computer model which will be used to prepare an individual to navigate an actual piece of terrain. 


AUA UN m 


Buildings 
Buildings 
Buildings 
Buildings 
Buildings 


Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 
Misc 


Terrain 
Terrain 
Terrain 
Terrain 
Terrain 
Terrain 
Terrain 


factory 

houses 

public buildings 
shacks 

other 


compass 
road signs 
rock piles 
sand bags 
street signs 
the sun 
people 
animals 
sound 
other 
other 


clearings 
depressions 
hills 

knolls 
ridgelines 
spurs/fingers 
other 
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roads 
roads 
roads 
roads 
roads 


obstacles 
obstacles 
obstacles 
obstacles 
obstacles 
obstacles 
obstacles 


vegetation 
vegetation 
vegetation 
vegetation 
vegetation 


water 
water 
water 
water 
water 
water 


dirt roads 
foot paths 
paved roads 
trails 

other 


electric lines 
pits/fox holes 
shallow ditches 
telephone poles 
towers 
trenches 

other 


bushes 
flowers 
grass/weeds 
trees 

other 


lakes 

marsh lands 
ponds 
puddles 
swamps 
other 


APPENDIX F. COURSE 


1. GENERAL 

This appendix consists of six items: the 1:24,000 scale map enlarged to 1:5,000 
scale, an aerial photo of the course, an aerial photo with an example participant 
debriefing route, 1:5,000 course orienteering map, and an explanation of the map legend 
[BANK 97]. The 1:50,000 and 1:24,000 maps are the standard scales used by most US 
ground forces for military operations. The difference in detail between the 1:5,000 scale 
map and the 1:24,000 map are obvious upon comparison. The aerial photo is the same 
given to the low fidelity map subjects. The 1:5,000 scale orienteering map was created by 
Banker [BANK 97] and modified by Goerger [GOER 98b]. The map legend explanation 
is taken directly from Appendix D of MAJ Banker’s 1997 Masters Thesis. 
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2. 1:24,000MAP EXCERPT OF COURSE AREA 
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Figure F.2. 1:24,000 Map Excerpt of Course Area 
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3. 1:24,000MAP ENLARGED TO 1:5,000 
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4. AERIAL PHOTO 





Figure F.4. Aerial Photo 
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5. AERIAL PHOTO WITH PARTICIPANT ROUTE 





Figure F.5. Aerial Photo With Subject Route 
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5,000 SCALE ORIENTEERING MAP 
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7. COURSE MAP LEGEND EXPLANATION 

All maps are generalizations. They use symbols to portray actual features on the 
earth’s surface. Not all features are represented with the same precision. Discrete non- 
vegetation items are plotted on the map in the exact location they are in the actual 
environment, whereas vegetation boundaries (unless indicated with a distinctive dotted 
line) are not meant to represent a clean break from one type of vegetation to ouem 
Rather, this line separating one vegetation area from another is a generalization of where 
one type more or less ends and another more or less begins. The line separating the two 
can best be thought of as a blurry line where the two types of vegetation intermingle. The 
below guide will help to determine the specific limitations of each symbol on the 
orienteering map. 

Building - Buildings in the area are of several types: 

a. Latrines - most common building, tan in color, approx. size 3 x 8 meters 

b. Shelters - second most common building, green wood, roofed, no walls, 

approx. size 3 x 8 meters 
c. Admin. - field office and shack, black with gold trim, 8 x 8 meters and 2 x 2 
meters respectively 

Open Sandy Ground - a significant patch of sand that will slow running 

Open ground - dirt, hard pack, free of grass and other vegetation. 

Undergrowth walk - immature chaparral or oak, dense stands of bushes, 
incomplete overlap of two distinct areas of fight which allow restricted passage along that 


overlap, other plants that prevent running. 


153 


Fight - mature chaparral or immature oak in such density that passage through is 
very difficult, running impossible 

Forest walk - oak forest with patchy undergrowth, low lying tree limbs or tree 
density that prevents running from being sustained 

Forest slow run - oak forest fairly free of undergrowth, but with low lying limbs 
or tree density that makes sustained running difficult. 

Rough open ground - grass covered ground, possibly with scattered (avoidable) 
undergrowth. Note that there are a few locations that have what appears to be old jeep 
trails but are portrayed as rough open ground. Sometimes the distinction between one or 
the other blurs. If in doubt refer to other more distinctive features (contour lines, etc.) to 
determine your location. 

Shallow depression - most likely an old decaying foxhole position or other man 
made excavation where the banks have eroded to create a bowl-like depression of 1 to 3 
feet below surrounding ground. 

Misc. object - a manmade feature, rubble, derelict military equipment, or other 
item whose exact description is only provided if it is the location of a control 

Pit - an old foxhole or likely other man made pit that has steep vertical walls and 
may be reinforced with wood, depth from 2 to 5 feet. Note that there will be many pits in 
the area that are not depicted on the map. The pits that are depicted are accurate. 

Telephone poles - wood poles (if bearing wire it will be noted on map) approx. 
25 to 30 feet in height 

Concrete pad - old concrete tent pad extending from 2 to 5 inches above ground 


level 
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Tree - a tree or large bush (could be two or more trees growing close together — 
forming an unbroken single canopy -- if the trees are small) 

Rootstock - a dead or overturned tree 

Troop training device - a bunker or other man made item built for training 
soldiers 

Vegetation boundary - the edge of a vegetation type 

Gully or Ditch - ranging from a shallow 1-foot deep gully to 5-foot deep military 
trench 

Jeep Trail - a road more suitable for 4 x 4 vehicles due to width restriction and/or 
ruts. May be distinctive and worn or in some places overgrown with grass but still 
containing ruts. 

Paved Road - a surfaced all weather road 

Road - a sandy or dirt road wide and level enough for 2 wheel drive vehicles 

Indistinct Path - a path that is in the process of being overgrown with only 
intermittent marks on the ground that indicate that it was once a well traveled path 

Narrow Ride - a linear break in the forest that may have once been a jeep trail 
but now is overgrown with grass and lacks telltale wheel ruts 


Path - a foot or bike path. 
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8. CLUE SHEET 


Course Orange Length 2070 meters 


Building 


Pit Shallow 


Small depression Shallow 
Single tree Deciduous Height 4 m 
Ruin Size 3x7 


Dry ditch Ruined 


Clearing 
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APPENDIX G. PARTICIPANT TASK LIST’ 


Thank-you for participating in this study. You will do an Orienteering course 


today. However, there are some important differences to note: 


1. You will be wearing a light pack with DGPS and Newton MSG Pad 130. Its purpose 
is to log your route and act as a data capture device for other actions you may perform. 


2. Before you run the course you will carefully plan your route through the entire course 
(see Important Information on Marking Your Map) 


3. Use this training time to commit the route and course to memory. You are expected to 
do the following on the actual course run: 


a. Navigate without aid of map and compass, utilizing only your memory 
b. Attempt to find all the controls utilizing your planned route 


Summary of objectives 
All Objectives are equally important!! 


Choose the most efficient route based on your abilities 

Minimize the number of map checks you request from the administrator 
Minimize the number of compass checks you request from the administrator 
Minimize the number of map with compass checks you request from the 
administrator 

5. Stay on your planned route 

6. Find all the controls in order (you have 60 minutes to conduct this task) 


= 


If you need to make a map check then say so and the administrator will give you the 
map for (30/60) seconds. Additional time can be requested in (30/60) second 
increments at the additional cost of a map check each. 

If you need to make a compass check then say so and the administrator will give you 
the compass for 30 seconds. Additional time can be requested in 30-second 
increments at the additional cost of a compass check each. 

If you need both map and compass then say so and the administrator will give you 
both for (60/90) seconds. Additional time can be requested in increments of (60/90) 
seconds. 

If you want to change your route announce to the administrator that you are changing 
your route plan. At that point the administrator will hand you the map, compass, and 
blue pen. From the time that he gives you the materials you will have 60 seconds to 
plot the new route. If you need more time then tell him you need more time and you 
will get another 60 seconds. Request additional time as needed but remember that one 
of your objectives is to make as few map checks as necessary. Every (30/60) seconds 


! This document is adapted and modified from MAJ Banker's Masters Thesis [BANK 98] 
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that you are looking at the map beyond the original 60 seconds for the route 
change counts as a map check. 


Note that when two times are given (30/60) the larger time applies to low fidelity map 
subjects since they had both the map and the aerial photograph to consult. 
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APPENDIX H. MAP MARKING INSTRUCTIONS? 


Pay close attention to how you mark your route, be as precise as the map and pen 
allow. Before your actual run you are expected to preview your map within your group’s 
prescribed context. Mark your planned route using the RED pen. You may correct any 
mistakes you make while planning with the white eraser. Once the planning period is up 
or you elect to finish you will not be allowed to erase any of the red route marks you have 
made. SO BE PRECISE in marking your map, detail does matter. Later during the 
actual course run anytime that you are going to deviate from your planned route you must 
stop: 


. Announce to the administrator that you are changing your route plan. At that point the 
administrator will hand you the map. From the time that he gives you the map you will 
have 60 seconds to plot the new route. If you need more time than tell him you need more 
time and you will get another (30/60) seconds. Request additional time as needed but 
remember that one of your objectives is to make as few map checks as necessary. Every 
(30/60) seconds that you are looking at the map beyond the original 60 seconds for 
the route change counts as a map check. 


. Take the blue pen and draw in your new route with the same attention to detail that you 
applied or the original route planning in red. 


. Leave your original route on the map. The eraser is provided so that you may make 
corrections to a route as you draw it. Once you finish drawing and begin navigating you 


are not allowed to erase routes, or corrections to planned routes (blue penned routes). 


. You may make as many corrections to your route(s) as necessary while navigating the 
course. 


Importance of detail in map marking and navigation 


You are allowed to deviate from your planned route within the following 
tolerances while still being considered on that route: 


Jeep Trails, Paved Roads, Unpaved Roads, Indistinct Paths, Narrow Rides and 
Paths -- If your marked route is on any of these features you are allowed 5 meters either 
side of the feature and you are still considered as being "on your route”. 

All other features -- On all other types of non road/trail terrain you may travel 15 


meters to either side of your marked route and you are still considered as being "on your 
route" 


l'This document is adapted and modified from MAJ Banker's Masters Thesis [BANK 98] 
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APPENDIX I. DIGITAL PHOTOS 


1. GENERAL 


As part of their training aids, subjects are provided with a series of digital images 
of the control points. Map-only subjects received the photos displayed in Appendix 1.2 
while Virtual Environment subjects receive the Appendix I.3 photo sets. The VE subjects 
also received screen shot images from the VE of what the control looks like in the model. 
This is so the subject can compare the model to the real world. The photos are furnished 
in color. Having the corresponding VE screen shot helps VE subjects overcome some 
model deficiencies, such as lack of accurate ground cover, and lack of negative spot 
elevations (i.e., pits and ditches) in the model. The photographs provide a useful tool for 
the subject in fixing exactly what the target control point looks like, and perhaps some of 
what the surrounding terrain looks like. Having this type of information is not entirely 
unreasonable for a military mission. The photos help to outfit the subject with a stronger 


grasp of the defining landmarks they are searching for. 
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2. MAP AND REAL WORLD GROUP PHOTOS 





ontrol Point 2 Control Point 5 





Control Point 3 Control Point 6 
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3. VIRTUAL ENVIRONMENT GROUP PHOTOS 
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Real World 
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Real World Model 
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APPENDIX J. COURSE EQUIPMENT CHECKLIST 
Binder Containing: 


Subject's map & designated route 
Think Out Loud Instructions 
Data Collection Sheet 


Researcher's Script 


Data Recording: 


blue alcohol pen to record route deviations 
red pen to record data 
digital camera 


helmet & 8mm camera 


rucksack frame w/GPS system 


stop watch/timer 


Misc: 


____ extra battery (8mm camera) 

. . .. extra cassette (8mm camera) 

.  .... extra Color Wheels for Tasks 3.1. & 5.1 
______ extra arrows (color wheels) 

______ extra clue sheet (incase subject looses his/hers) 
_____ blindfold (for movement to course) 

____ cellular phone (optional) 
____ compass 

____ first aid kit 

Tecnu (for poison oak) 


water 


Prepositioned: 


Color Wheel Platform for Tasks 3.1. & 5.1 
Control flags 
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APPENDIX K. THINK OUT LOUD INSTRUCTIONS! 


Your thoughts are important to this research. As you navigate the course, you 
should be "thinking out loud". 


As you move through the environment and experience it directly, express what 
you are thinking. The mental preconception you had of this environment before you 
stepped into it will now be evaluated by you as you experience the course directly. As 
this image is confronted with direct experience your expectations and plan may be 
confirmed, modified, or refuted. Be sure to talk “out loud" these thoughts. 


The process of talking “out loud” and paying close attention to your route will 
slow you down. This is expected and why you are given an hour to finish the course. 


PLEASE SPEAK LOUDLY SO THAT YOUR VOICE WILL BE PICKED UP BY 
THE MONITOR AND HE CAN RECORD WHAT YOU SAY. 


! This document is adapted and modified from MAJ Banker's Masters Thesis [BANK 98] 
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APPENDIX L. ROUTE CLASSIFICATIONS 

1. GENERAL 

This appendix consists of five items: route analysis, an explanation of route 
classifications for each leg of the course [BANK 97], and route classifications based on a 
Goerger’s LISP Program [GOER 98b]. The explanation of route classifications for each 
leg of the course is taken directly from Appendix F of MAJ Banker’s 1997 Masters 
Thesis. The explanation of the LISP routes is taken directly from MAJ Goerger’s 1998 
Master Thesis. Route classifications were utilized to categorize the difficulty of an 
individual’s planned routes for comparison to their navigational ability. Routes were 
classified using MAJ Banker’s route classification listing and again utilizing the results of 


Goerger’s LISP route planning program. 


2. ROUTE ANALYSIS 

Participant routes were analyzed for difficulty level and performance. 
Participants’ Leg Error Scores were correlated with their Leg Difficulty Rating and 
ability level. The relationship between a subject’s route difficulty and their overall 
performance is discussed in Chapter IV, Section B.3. 

Goerger felt that if participants planned routes which were beyond their ability 
level (a novice planning expert routes), the chances they will fail to successfully execute 
the planned routes increases. Advanced routes, while being faster and perhaps more 
direct, required true expertise in navigational skills to successfully implement them. In 
fact, most “expert” navigators did not plan “expert” routes. The poorest routes were 


planned by novices who failed to realize what exactly they were attempting. 


3. BANKER’S ROUTE CLASSIFICATIONS 

What follows is MAJ Banker’s classification of some of the most probable routes 
to a given control and is based on the International Specification for Orienteering Maps 
[INTE 90]. They do not represent the only ways of getting to a control but the most likely 
routes chosen by participants based upon MAJ Banker’s orienteering experience and 
knowledge of the terrain. The classifications are used as a basis for comparison with the 


routes selected by Goerger’s LISP Route Selection Program. Note that all controls 
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possess at least one beginner’s route. If a participant did not exactly follow the route 
described by Banker, the route was examined. If it had numerous handrails, it was 
considered intermediate. If it had more catching features, it was considerd advanced. A 
lack of any navigational features, or a direct line azimuth between any but the closest 
points earned an advanced rating [GOER 98b]. 
a. Control 1. 
li Beginner 
a) Gigling Road west to jeep trail 


b) Jeep Trail south by east by south to building 
c) Control on NW corner of building 


2; Верїппег 


а) Watkin's Gate Cutoff to indistinct path. 

b) Indistinct path southwest up hill to jeep trail 
С) Jeep Trail west to building 

d) Control on NW corner of building 


3. Intermediate 


a) West through plotted individual trees (catching features) 

b) Handrail rough open ground south to junction indistinct 
path and jeep trail 

с) Jeep Trail west to building (catching feature) 

d) Control on NW corner of building 


4. Advanced 


a) West through plotted individual trees 

b) Follow runnable forest southwest 

с) Try to hit small rough open gap by keeping walkable forest 
to left shoulder 


d) Use forest fight to west as catching feature if needed 
e) Control on NW corner of building 
f) Use jeep trail for catching feature if control is missed 


5. Advanced 
a) Go straight at control from start 


b. Control 2. 


72 


с. 


d. 


| Beginner 


a) Jeep trail northwest to building 
b) Follow open ground to west and look for rough open 
clearing going northwest (handrail) 
с) Follow rough open clearing northwest looking for pit 
d) Control in pit 
2. Intermediate 
a) Jeep trail northwest to building 
b) Go straight at control (WSW) from building 
3 Advanced 
a) Set out on straight line directly for control 
b) Hit open ground and look for building on the right and 
rough open break on the left. (Catching feature) 
c) Follow rough open clearing northwest looking for pit 
d) Control in pit 
Control 3. 
ІС Beginner 
a) Head northwest and get out onto Gigling Road 
b) Take Gigling Road west to jeep trail junction with 
telephone pole 
с) Take jeep trail southeast to convergence of two jeep trails 
d) Head southwest into tree grove looking for control 
(1) Use building as catching feature 
(2) | Use open ground to west as backup catching feature 
e) Control hanging from tree limb 


27 Advanced 


a) 


b) 


с) 


Control 4. 


Head straight at control; use jeep trail prior to control as 
catching feature 
Head southwest into tree grove looking for control 


(1) Use building as catching feature 
(2) Use open ground to west as backup catching feature 


Control hanging from tree limb 
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| Вершпег 


а) Head southwesterly and try to get on jeep trail headed in 
same direction 
b) Take jeep trail to junction 
с) Take jeep trail southeast to junction 
d) Take southerly fork to next junction 
e) Take fork to northwest 
f) Once beyond patches of fight leave trail and start looking 
for control 
g) Control is in pit 
2. Beginner 
a) Turn around and go back to jeep trail to the east 
b) Take jeep trail southwest to junction 
с) Take fork to the south to another junction 
d) Take fork to the west to next junction 
e) Take southerly fork to next junction 
f) Take fork to northwest 
g) Once beyond patches of fight leave trail and start looking 
for control 
h) Control is in pit 
ЗЇ Intermediate 
a) Go south towards road junction 
b) Get on road and take to junction 
с) Take road west to other road junction 
d) Handrail around fight to west coming down through small 
patch of fight into control 
4. Advanced 
a) Head straight at control expect to hit jeep trail that runs 
NW to SE (catching feature) 
b) Hit trail and then thread way through scattered fight 
с) Emerge into center of depression and rough open ground, 
(catching feature) look for pit 
d) Control is in pit 
Control 5. 
1. Beginner 
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а) Move back out onto jeep trail 
b) Take trail west to trail junction 
с) Take trail WNW up to misc object 


d) From misc. object go straight at control 
2: Intermediate 
a) Move directly at control 
b) Use Gigling Road as catching feature if miss on control 
с) Control is in center of clearing 
2. Advanced 
a) Move directly at control 
b) Use southwesterly linear clearing as catching feature 
с) Follow clearing NW right into control 
d) Use Runnable forest along Gigling as catching feature in 
case of miss 
Control 6. 
@ Beginner 
a) Move out onto Gigling Road and take it westerly to 
junction with dirt road 
b) Move down dirt road (south) to junction with jeep trail 
с) Take jeep trail to east look for concrete rubble 
d) Move southeast through runnable forest 
е) Look for control on concrete pad 
2. Beginner 
a) Move straight at control and hit jeep trail 
b) Go southwest on Jeep trail to junction with another jeep 
trail 
с) Take jeep trail westerly and look for concrete rubble 
d) Move southeast through runnable forest 
e) Look for control on concrete pad 
де Intermediate 
a) Move south to junction of two jeep trails (catching feature) 
b) Handrail jeep trail southeasterly to clearing (catching 
feature) 
с) Handrail clearing to the west 


d) Hit fight going west (catching feature) and move south 


Iy5 


е) Handrail fight (keeping it on right shoulder) into control 
f) Look for control on concrete pad 
4. Advanced 

a) Move straight at concrete rubble (aiming off technique) use 
jeep trail as catching feature and handrail 

b) Move southeast through runnable forest 

c) Look for control on concrete pad 

g. Control 7. 
ү Beginner 

a) Move back out onto east west jeep trail 

b) Go west to junction of jeep trail and dirt road 

с) Take dirt road south to junction with four jeep trails 

d) Take jeep trail east by northeast 

e) Look for second linear break in vegetation (indistinct path) 

f) Take indistinct path (handrail) to ditch 

g) Follow ditch to its end 

h) Control at east end of ditch 

2 Intermediate 

a) Move through rough open ground easterly to jeep trail 
(catching feature) 

b) Follow jeep trail (handrail) to junction with other jeep trail 
by building 

с) Locate telephone poles and follow wire (handrail) south 
easterly 

d) Hit fight and turn west and follow fight boundary into ditch 
(handrail) 

e) Control at east end of ditch 

3 Advanced 

a) Move through rough open ground easterly to jeep trail 
(catching feature) 

b) Take jeep trail to curve where it turns east (hand rail) 

c) Leave jeep trail and head straight for control use east west 
jeep trail as checkpoint (catching feature) 

d) Aim off to east side of ditch and go southeast (telephone 
wires to east as catching feature to prevent drifting too far 
east) 

e) Use fight as catching feature 
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f) Hit fight and turn west and follow fight boundary into ditch 
g) Control at east end of ditch 


4. Advanced 
a) Move straight at control 
b) Use jeep trail junction as attack point 
с) From attack point take offset route to west part of ditch 
d) Follow ditch to east and find control at end of ditch 
Control 8. 
1. Beginner 
a) Handrail fight to the east till hitting the jeep trail 
b) Follow jeep trail northerly through intersection to sharp 
curve to the east 
с) Once at sharp curve to east turn off trail to west and look 
for control in clearing 
d) Control located in clearing 
2: Intermediate 
a) Handrail fight to telephone poles 
b) Take telephone poles NW back to jeep trail junction 
с) Follow jeep trails east to next junction 
d) Take jeep trail north 
e) Leave jeep trail and move directly at control 
3i Advanced 
a) Move directly at control (avoiding forest walk) use jeep 
trail junction as catching feature 
b) From jeep trail junction aim off to east of control at sharp 
curve to east of jeep trail keeping eyes open for control in 
clearings 
C) Use same trail as Beginner route as catching feature (for 
drift) 
Control 9 (Finish) 
1. Beginner 
a) Move back out to jeep trail just to east of control 8 
b) Take trail south to four way junction with other trails 
(handrail) 
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с) Take southeasterly running trail to trail fork 


d) Take northeasterly running fork to five way junction 
(handrail) 

e) Take northwesterly running trail keeping eyes open for 
small break in fight to the east (catching feature) 

f) Take indistinct path into clearing and hook to north 

g) Control on east edge of clearing 

2. Intermediate 
a) Move back out to jeep trail just to east of control 8 
b) Move off trail using rough open to move closer to control 


с) Take rough open out onto jeep trail which runs NE to SW 
d) Take trail to junction with North South jeep trail 

e) follow jeep trail looking for indistinct path 

f) Take indistinct path into clearing and hook to north 

g) Control on east edge of clearing 


3. Advanced 


a) Move straight at control on east by northeast azimuth 

b) Use trail as catching feature 

с) Fight to north and south of route used as catching features 
d) Locate opening in fight 

е) Take indistinct path into clearing and hook to north 

f) Control on east edge of clearing 


4. LISP PROGRAM ROUTE CLASSIFICATION 

This section is taken directly from MAJ Goerger’s thesis [IGOER 98b]. This LISP 
program plans a route through a specified piece of terrain based on identifiable decision 
points and terrain characteristics. The information is manipulated by a branch and bound 
search, pruning heuristics, and terrain classification. 

The program is designed to locate three optimal paths through the course. One 
Beginner (Figure L.1), one Intermediate (Figure L.2), and one Advanced Course (Figure 
L.3) are calculated and displayed on maps for comparison with participant maps. The 
program also produces a sequential list of decision points or waypoints to traverse in 
order to complete the course. Each leg of a participant’s route is compared to the LISP 
program route legs. If two LISP routes have legs that are the same, the leg is classified as 
the easier of the two routes. If a participant's planned route between control points is not 


the same as any of the computer program's planned routes, the participant's route is 
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assigned a classification which is most closely associated with the participant’s route with 


respect to the program algorithm’s defining characteristics. 


& Fart Ord Land Navigation Мар 





Figure L.1. LISP Beginner Route 
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Figure L.2. LISP Intermediate Route 
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Figure L.3. LISP Advanced Route 
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APPENDIX M. DATA COLLECTION WORKSHEETS 


1. TRAINING PHASE DATA COLLECTION SHEET 
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е еен PHASE DATA COLLECTION SHEET 
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APPENDIX N. PARTICIPANT DATA 
1. GENERAL 
Subject data consists of two items: map with planned route, and the map with 
executed route. For subjects who had the low fidelity map, their planned and executed 
routes are shown on the overhead photo. The errors for deviation from the planned route 
are located in Appendix O. The angle and distance measurements for the Wheel and 


White Board Tests can be found in Appendix O.2 and O.3 respectively. 
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Figure N.1. MI1-1 Planned Route 
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Figure N.2. M1-1 Executed Route 
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Figure N.3. M1-2 Planned Route 
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Figure N.4. M1-2 Executed Route 
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Figure N.5. M1-3 Planned Route 
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Figure N.6. M1-3 Executed Route 
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Figure N.8. M1-4 Executed Route 
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Figure N.9. M1-5 Planned Route 
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Figure N.10. M1-5 Executed Route 
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Figure N.21. V1-1 Planned Route 
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Figure N.22. V1-1 Executed Route 
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Figure N.23. V1-2 Planned Route 
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Figure N.24. V1-2 Executed Route 
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Figure N.25. V1-3 Planned Route 
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Figure N.26. V1-3 Executed Route 
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Figure N.27. V1-4 Planned Route 
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Figure N.28. V1-4 Executed Route 
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Figure N.29. V1-5 Planned Route 
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Figure N.30. V1-5 Executed Route 
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APPENDIX O. RAW DATA 


1. GENERAL INFORMATION 

Participant data is referenced by the participant identification (ID) label ( M1 — Train-to- 
Standard Map-Only Group, M2 — Low Fidelity Map Map-Only Group, VEI - Train-to- 
Standard VE, VE2 — Low Fidelity Map VE). The number corresponds to the 
participant’s internal group label. Data fields that are left blank represent information not 


recorded because a participant did not undergo the test or failed to reach that point in the 





SS id 
22-Apr-99 
\1-2  VE1 29-Apr-99 11:48 31 


O3 Army AM Intermediate 


M 
M 
M 
M 
M 
M 
M 
F 
M 
F 03 
M 
M 
M 
M 
M 
M 
M 
M 
M 


Mi-1  MAP1 23-Apr-99 9:22 32 O3 Marine SC Intermediate 


М1-2  MAP1 30-Apr-99 9:30 31 O3 Army INF Expert 
V1-3  VE1 7-May-99 10:00 32 O4 Army AR Intermediate 
М1-3 MAP1 13-May-99 12:00 36 O4 Army ADA Expert 
Vi-4 МЕ1 14-May-99 13:00 32 O3 Marine SC Intermediate 
\1-5 \МЕ1 15-May-99 10:00 30 O3 Army SC Intermediate 


М1-4  MAP1 21-May-99 9:00 32 ОЗ Army SC Intermediate 


М1-5 MAP1 5-Jun-99 8:00 36 O3 Marine SC Expert 
V2-1 VE2 11-Jun-99 9:00 30 Marine SC Intermediate 
M2-1  MAP2 3-24-99 6:30 32 O3 Marine INF/FIN Intermediate 
V2-2  VE2 13-Jul-99 8:00 29 O3 Marine SC Intermediate 
M2-2  MAP2 15-Jul-99 13:30 31 O3 Army ADA Intermediate 
V2-3 VE2 16-Jul-99 7:00 40 O4 Marine SC Intermediate 
M2-3 MAP2 16-Jul-99 12:00 34 O3 Marine SC Expert 
V2-4 VE2 17-Jul-99 11:00 30 O3 Army AR Intermediate 
M2-4  MAP2 19-Jul-99 13:00 32 
\2-5 МЕ2 21-Jul-99 12:00 35 


M2-5 МАР2 23-Jul-99 8:00 33 


ОЗ Army АУ Expert 
O3 Marine SC Intermediate 
O3 Army MI Expert 
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2. INITIAL TESTES AND QUESTIONNAIRE RESULTS 


The initial tests and questionnaires are in Appendix E. Answers for the Map Test 


are located in Appendix E.1. 


yey | А => 
а БАГ 



















Intermediate 





17.5 
Mi-2 15.5 Low 19 Expert 
Mi-3 8.25 Low 17 Intermediate 
Mi-4 24.25 High 19.5 Intermediate 
Mi-5 21.5 High 18.5 Expert 
M2-1 27 High 19 Intermediate 
M2-2 42.25 High 18 Expert 
M2-3 56.25 High 18 Beginner 
M2-4 20.75 High 19 Intermediate 





M2-5 14.5 Low 18.5 Intermediate 
V1-1 23 High 19.5 Intermediate 
V1-2 42.25 High 20 Intermediate 
V1-3 10.5 Low 19 Intermediate 
\1-4 19.5 Low 19.5 Intermediate 
\1-5 35 High 19.5 Intermediate 
V2-1 10.75 Low 17 Beginner 

V2-2 29.25 High 17.5 Intermediate 
V2-3 31.25 High 19 Intermediate 
V2-4 14.25 Low 18.5 Intermediate 
V2-5 23 High 17 Intermediate 











3. ROUTE ERRORS 

The data provided in this section consists of the map checks, errors, error 
distances, and route leg classifications. The data appears in its raw form, summations, 
and normalized form for each of the experiment’s twenty participants. The abbreviations 


utilized are listed in Table O.1. 
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Table O.1. Route Errors Abbreviation Table 


a. Route Data Summation 
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4. Route Data Leg CP2 to CP3 
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h. Route Data Leg CP6 to CP7 
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j. Route Data Leg CP8 to CP9 
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m. Leg Difficulty Evaluation Banker 
Each leg is evaluated utilizing MAJ Banker's Route Classification (Appendix 
1.2). “В” stands for Beginner, “T”’ stands or Intermediate, and “A” stands for Advanced. 
The total is based on a point value system of B = 1,1 =2, and A = 3. The Average is the 
total divided by the number of legs. 0-1.50 is an average course difficulty of Beginner, 
1.51-2.5 is Intermediate, and 2.51-3.0 is Advanced. The same criteria apply to the Lisp 


leg difficulty evaluation. 
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WHEEL TEST RESULTS 
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a. Wheel Test Results for Control Point 2 


С | р 
| : 
567 t 
210 

- Ei EE lo 
ie Я T о 
1 N 
| | ; В 
Iw = 

3 E à 
D = 

"1o N 


140 


106 |_з10 | 165 | 30 | 70 | 5 | 35.00 | Nw - 
[во [210 | 1з0 | -20 | 30 | 4 | 30.00 | sw | 55 — 
[60 | 2233 | izo | 10 | : | 50 | 2567 | South | 60 — 


V1-1 
V1-2 
V1-3 
V1-4 
V1-5 





[9o | 250 | 140 | -0 | -10 | 30 | 20.00 | west | 58 


b. Wheel Test Results for Control Point 4 and Total Wheel Test Angular Difference = 


| мек | 45 | 45 T 22057 ни 32.167 | 


ошо ож ош esa asco [aa ae и тв И 


Бей ТЕК ШТІ | еза" 


М1-1 
М1-2 


~ 
Ф 
E 
т 
ш 
e 
: 
~ 
a 
e 
са 
^ 
m 
à 
d 
m 
5 
о 
g 
+ 
R 
~ 
N 
e 
à 
++ 
: 


à 


1 


"TU 


N 
E 
$ 


у 
| 
| 
| 
E 
| 
1 
| 
| 
| 
қ 
1 


Tin [с 
HE TE i 


| 


| 
|< 
| 
| 


| 


: —— — 
5 | " 
| БР ЧЕ » МР“ 


Ы} 


j T 5 г 
СЕ ЕЕЕ E 


—À— nam 


———— P nel 


E 


~ * nno 
= - а а 46. | | 


2 
Св | 





236 


5. WHITE BOARD RESULTS 
a. White Board Normalized Distance Differences from Actual Normalized 


Distances 
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b. White Board Angles 
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6. UNPLANNED ROUTE RESULTS 
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APPENDIX P. NAVIGATION TERMS 


The following navigational terms are used in this thesis. This appendix is included 


to explain the terms in more detail than is allowed in the thesis body. 


HANDRAILS - A linear feature that runs parallel to the route the subject is following. 
This can be used to provide directional cues. For example, a subject may choose a route 
which handrails a creek. As long as the subject can see the creek to his side, he knows he 


is On course. 


CONTROL POINT - This is the marker which is what orienteerer/student has to find 
when taking part in a land navigation course. This marker may be a flag, post, sign, or 


other device which marks the target location. 


CHECKPOINTS — A prominent point that you can be sure to identify, both on the ground 
and on the map. These are sometimes known as reference points. Military navigators 
using dead reckoning are advised to dead reckon between interim checkpoints as opposed 


to trying to do it in one step. 


CATCHING FEATURES - A linear feature behind or beyond your intended destination. 
Much like a backstop in baseball, this is meant to stop you and let you know that you 


have gone too far. 


NAVIGATIONAL CORRIDORS - This is having a handrail to either side, preventing 
you from straying too far to the left or right. Combined with a catching feature, this 


*boxes in" the destination. 


ATTACK POINT - This is used when the desired objective is small or may be difficult 
to locate. The attack point is an easily recognizable checkpoint near the objective. You 
first locate the attack point, then plan a precise movement to the objective. If you miss 


finding the objective, you merely return to the attack point and try again. 


24] 


EXPANDED OBJECTIVE - Similar to an attack point, this is used when the objective is 
part of a larger terrain feature, say a hilltop. This simplifies the problem as you first 


locate the larger terrain feature (i.e., the hill), then continue on to the actual objective. 


ROUGH COMPASS - This involves using a compass to establish a general direction of 
travel to the next destination area or checkpoint. This is useful if the next target area is a 


road or other linear feature. 


BOX AROUND - This technique is used when following a dead reckoning azimuth and 
a significant obstacle is encountered. The subject would turn at 90 degrees to the 
direction of travel and proceed at a measured distance (the box pace count) until the 
obstacle was clear to his side. The subject then continues parallel to the line of original 
travel resuming the original pace count until the obstacle was cleared again. Now the 
subject turns 90 degrees in the opposite direction, repeats the original measured distance 
(the box pace count), and then returns to the original direction of travel at the pace count 
they had at the end of the parallel leg. Difficulties arise when the obstacle is not uniform 
in size. Having to maintain two different pace counts can also be difficult for some 


individuals. 
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